System and method for enhanced training using a virtual reality environment and bio-signal data

ABSTRACT

A training apparatus has an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user. The user interacts with content that is presented in the VR environment. The user interactions and bio-signal data are scored with a user state score and a performance scored. Feedback is given to the user based on the scores in furtherance of training. The feedback may update the VR environment and may trigger additional VR events to continue training.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims all benefit, including priority, of U.S.Provisional Patent Application Ser. No. 62/048,923, filed Sep. 11, 2014and entitled SYSTEM AND METHOD FOR DETERMINING MENTAL RESPONSE TO AVIRTUAL ENVIRONMENT, the contents of which is incorporated herein byreference, in its entirety.

FIELD

Embodiments described herein relate to wearable devices. Embodimentsdescribed herein relate more particularly to wearable devices withbio-signal sensors for training.

BACKGROUND

A computing device may include a display to provide visual output datafor the user. A virtual reality (VR) environment may be displayed on thedisplay to provide a computer simulation of real world elements. A usermay provide input to a computing device for example using a keyboard,mouse, track pad, touch screen, or motion-capture devices.

A human brain generates bio-signals such as electrical patterns known,which may be measured/monitored using an electroencephalogram (“EEG”).These electrical patterns, or brainwaves, are measurable by devices suchas an EEG. Typically, an EEG will measure brainwaves in an analog form.Then, these brainwaves may be analyzed either in their original analogform or in a digital form after an analog to digital conversion.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, withreference to the attached figures, wherein:

FIG. 1 is a perspective view of an implementation of an example deviceto provide a VR environment according to some embodiments;

FIG. 2 is a schematic view of an implementation of an example systemaccording to some embodiments;

FIG. 3 is a schematic view of an implementation of another examplesystem according to some embodiments;

FIG. 4 is a flow diagram of an implementation of a process that may beused to implement aspects of the systems of FIG. 1, FIG. 2 and FIG. 3according to some embodiments;

FIG. 5 a schematic of an implementation of determine feedback operationusing Tables 5A and 5B according to some embodiments;

FIG. 6 illustrates a side view of an exemplary application according tosome embodiments;

FIG. 7 shows an example visual representation of content and/or feedbackrelating to 3D brain activity in a VR environment according to someembodiments;

FIG. 8 illustrates an example of a user interaction with the trainingsystem according to some embodiments;

FIG. 9 shows an exemplary application for fostering an emotionalconnection in a VR environment according to some embodiments;

FIG. 10 shows an exemplary application for representing emotionalavatars in a VR environment according to some embodiments;

FIG. 11 shows an exemplary application for a therapeutic VR petaccording to some embodiments;

FIG. 12 shows an example user interface for architectural design or gamedesign to generate or update aspects of the VR environment according tosome embodiments;

FIG. 13 shows example content of a VR environment of a user who isworking and who desires to focus on what he is doing as an aspect oftraining according to some embodiments;

FIG. 14 shows a schematic of an example magnification and extra detailsof a user focusing on an area of interest according to some embodiments;

FIG. 15 illustrates a schematic view of an implementation of a computerdevice.

FIG. 16 illustrates a schematic view of an implementation of anotherexample system according to some embodiments.

In the drawings, embodiments of the invention are illustrated by way ofexample. It is to be expressly understood that the description anddrawings are only for the purpose of illustration and as an aid tounderstanding, and are not intended as a definition of the limits of theinvention.

DETAILED DESCRIPTION

Reference will now be made in detail to some specific examplesembodiments including the best modes contemplated by the inventors forcarrying out the invention. Examples of these specific embodiments areillustrated in the accompanying drawings. While the invention isdescribed in conjunction with these specific embodiments, it will beunderstood that it is not intended to limit the invention to thedescribed embodiments. On the contrary, it is intended to coveralternatives, modifications, and equivalents as may be included withinthe spirit and scope of the invention as defined by the appended claims.Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of illustrative description andshould not be regarded as limiting.

For example, the techniques and mechanisms of various embodiments aredescribed in the context of particular tasks or functions in thetraining of humans. As the ways in which humans interact with computingdevices change, computers may become usable for new purposes or may bespecifically configured to be more efficient in performing existingtasks. Embodiments described herein may involve measuring and analyzingbio-signals such as brainwave patterns for a variety of practicalapplications including improving the training of humans for performingcertain tasks or functions. It should be noted that the techniques andmechanisms of the embodiments described herein apply to a variety ofdifferent tasks or functions in the training of humans.

Embodiments described herein may provide an integrated computing deviceor system with an interactive VR environment with visual output data toprovide a computer simulation of physical elements for training. A usermay interact with the VR environment using various input devicesincluding bio-signal sensors and in response the VR environment mayupdate or modify to provide enhanced training applications.

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention.Particular example embodiments may be implemented without some or all ofthese specific details. In other instances, well known processoperations have not been described in detail in order not tounnecessarily obscure the present invention.

Various techniques and mechanisms of the embodiments described hereinwill sometimes be described in singular form for clarity. However, itshould be noted that some embodiments include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. For example, a system uses a processor in a variety ofcontexts. However, it will be appreciated that a system can use multipleprocessors. As an illustrative example, a wearable device may include aprocessor that may in turn connect to a client device with anotherprocessor. The wearable device, client device, or both, may also connectto a cloud server system with a processor. Furthermore, the techniquesand mechanisms of the embodiments described herein will sometimesdescribe a connection between two entities. It should be noted that aconnection between two entities does not necessarily mean a direct,unimpeded connection, as a variety of other entities may reside betweenthe two entities. For example, a processor may be connected to memory,but it will be appreciated that a variety of bridges and controllers mayreside between the processor and memory. Consequently, a connection doesnot necessarily mean a direct, unimpeded connection unless otherwisenoted.

According to an aspect of embodiments described herein, there isprovided training apparatus with an input device and a wearablecomputing device with a bio-signal sensor and a display to provide aninteractive VR environment for a user, the bio-signal sensor receivesbio-signal data from the user, the bio-signal sensor comprising abrainwave sensor. VR may provide a computer simulated experience thatreplicates, creates or enhances an environment that simulates physicalpresence in places in the real or non-real world. A VR environment mayalso refer to an augmented reality environment which may be a livedirect or indirect view of a physical, real-world environment whoseelements are augmented or supplemented by computer processes. A user mayinteract in that world using input data such as gesture data, manualinput, sensor data, and so on. The computing device having or incommunication with a processor configured to as part of the interactiveVR environment, present content on the display where the content has aVR event, desired user states, and desired effects; receive user manualinputs from the input device which have effects in the interactive VRenvironment including during the VR event. In response, the processorreceives the bio-signal data of the user from the bio-signal sensorduring the VR event and processes the bio-signal data to determine userstates of the user, including brain states, during the VR event, wherethe user states are processed using a user profile stored in a datastorage device accessible by the processor and the user states includebrain states. The processor determines a user state score by comparingthe user states of the user to the desired user states during the courseof the VR event and determines a performance score by comparing theeffects to the desired effects during the course of the VR event. Theprocessor provides feedback to the user where the feedback is based on acombination of the user state score and the performance score.

In some embodiments, the feedback may be provided as a visualrepresentation in the VR environment to have effects in the VRenvironment, including during the VR event to provide visual, real timeor near real time feedback. The feedback may be an additional VR event(as part of a sequence of VR events or otherwise) to restart thebio-signal acquisition and scoring operations and to provide additionalfeedback to the user.

In some embodiments, there may be multiple VR events provided as asequence of VR events as part of a training program. The sequence may bea dynamic sequence that may modify or vary based on the user state scoreor other aspects of the VR environment or user states.

In some embodiments, the wearable computing device has an inertialsensor and the bio-signal sensor further comprises a facial bio-signalsensor, and where the bio-signal data further comprises data from thefacial bio-signal sensor and the inertial sensor; the computing deviceis further configured to receive the bio-signal data from the facialbio-signal sensor and the inertial sensor for a user head and eye gazedirection, wherein the user states further comprises the user head andeye gaze direction, and the desired user states further comprises adesired user head and eye gaze direction.

In some embodiments, the brain states has one or more of ability ofoperator to learn; prediction error; and emotional state leading toimpaired thinking.

In some embodiments, the device presents the user state score and theperformance score synchronized with the content and the VR event toassist the user to better attain the desired user states and desiredmanual inputs on the input device.

In some embodiments, the computing device is further configured to:revise the content in response to the feedback provided to the userwhere the user is further trained on the revised content. In someembodiments, the revised content has further effects in the interactiveVR environment including during the VR event to update or modify the VRenvironment. In some embodiments, the revised content is a revised VRevent, revised desired user states, and/or revised desired effects toprovide an iterative and cyclical training process.

In some embodiments, the user state score further comprises failurebrain states.

In some embodiments, the display is a stereoscopic display.

In some embodiments, the apparatus has a second display for presentingthe content and a visual representation of the user states and manualinputs of the user in real time.

In some embodiments, the computing device having or in communicationwith the processor is further configured to: provide real time or nearreal time feedback to the user during the presentation of the content.The real time or near real time feedback may occur during the VR event.

In some embodiments, the VR event is associated with event time data anda portion of bio-signal data is associated with bio-signal time datacorresponding to the event time data, wherein the processor isconfigured to identify a portion of the bio-signal data based on theevent time data and process the portion of the bio-signal data todetermine the user states during the VR event, the bio-signal time datasynchronized to the event time data using a common timeline or clock ora synchronization or context operation.

In some embodiments, the VR event is associated with event time data andthe bio-signal data is associated with bio-signal time data, and whereinthe processor is configured to identify a time interval based on anexpected response time for the VR event and the event time data,identify a portion of the bio-signal data based on the time interval andthe bio-signal time data, and process the portion of the bio-signal datato determine the user states during the VR event, the bio-signal timedata synchronized to the event time data.

According to an aspect of embodiments described herein, there isprovided training apparatus with an input device, a display and awearable computing device with a bio-signal sensor to provide aninteractive virtual reality (“VR”) environment for a user. Thebio-signal sensor receives bio-signal data from the user, the bio-signalsensor comprising a brainwave sensor.

In some embodiments, the computing device having or in communicationwith a processor configured to as part of the interactive VRenvironment, present content on the display where the content has atraining program as a sequence of VR events, desired user states, anddesired effects; receive user manual inputs from the input device whichhave effects in the interactive VR environment including during one ormore VR events of the training program. In response, the processorreceives the bio-signal data of the user from the bio-signal sensorduring the VR event of the training program and processes the bio-signaldata to determine user states of the user, including brain states,during the VR event of the training program, where the user states areprocessed using a user profile stored in a data storage deviceaccessible by the processor and the user states include brain states.The processor determines a user state score by comparing the user statesof the user to the desired user states during the course of the trainingprogram and determines a performance score by comparing the effects tothe desired effects during the course of the training program. Theprocessor provides feedback to the user where the feedback is based on acombination of the user state score and the performance score. Thefeedback may be provided as a visual representation in the VRenvironment to have effects in the VR environment, including during theVR event to provide visual, real time or near real time feedback. Thefeedback may be an additional VR event (as part of a sequence of VRevents or otherwise) to restart the bio-signal acquisition and scoringoperations and to provide additional feedback to the user.

According to an aspect of embodiments described herein, there isprovided training apparatus with an input device and a wearablecomputing device with a bio-signal sensor, the wearable device inconnection with a display to provide an interactive virtual reality(“VR”) environment for a user. The bio-signal sensor receives bio-signaldata from the user, the bio-signal sensor comprising a brainwave sensor.The computing device having or in communication with a processorconfigured to as part of the interactive VR environment, present contenton the display where the content has a VR event, desired user states,and desired effects; receive user manual inputs from the input devicewhich have effects in the interactive VR environment including duringthe VR event. In response, the processor receives the bio-signal data ofthe user from the bio-signal sensor during the VR event and processesthe bio-signal data to determine user states of the user, includingbrain states, during the VR event where the VR event is linked to thebio-signal data based on timing data, where the user states areprocessed using a user profile stored in a data storage deviceaccessible by the processor and the user states include brain states.The processor determines a user state score by comparing the user statesof the user to the desired user states during the course of the VR eventand determines a performance score by comparing the effects to thedesired effects during the course of the VR event. The processorprovides feedback to the user where the feedback is based on acombination of the user state score and the performance score. Thefeedback may be provided as a visual representation in the VRenvironment to have effects in the VR environment, including during theVR event to provide visual, real time or near real time feedback. Thefeedback may be an additional VR event (as part of a sequence of VRevents or otherwise) to restart the bio-signal acquisition and scoringoperations and to provide additional feedback to the user.

In another aspect, there is provided a training method implemented usingan input device and a wearable computing device having or incommunication with a processor, a bio-signal sensor and a display toprovide an interactive virtual reality (“VR”) environment for a user,the bio-signal sensor receives bio-signal data from the user, thebio-signal sensor comprising a brainwave sensor; the training methodinvolving: as part of the interactive VR environment, presenting contenton the display where the content has a VR event, desired user states,and desired effects; receiving user manual inputs from the input devicewhich have effects in the interactive VR environment including duringthe VR event; receiving the bio-signal data of the user from thebio-signal sensor during the VR event; processing the bio-signal data todetermine user states of the user, including brain states, during the VRevent, the user states are processed using a user profile stored in adata storage device accessible by the processor and the user statesinclude brain states; determining a user state score by comparing theuser states of the user to the desired user states during the course ofthe VR event; determining a performance score by comparing the effectsto the desired effects during the course of the VR event; and providingfeedback to the user wherein the feedback is based on a combination ofthe user state score and the performance score.

In some embodiments, the wearable computing device further comprises aninertial sensor and the bio-signal sensor further comprises a facialbio-signal sensor, and where the bio-signal data further comprises datafrom the facial bio-signal sensor and the inertial sensor; the methodfurther comprising receiving the bio-signal data from the facialbio-signal sensor and the inertial sensor for a user head and eye gazedirection; wherein the user states further comprises the user head andeye gaze direction, and the desired user states further comprises adesired user head and eye gaze direction.

In some embodiments, the brain states comprises one or more of abilityof operator to learn; prediction error; and emotional state leading toimpaired thinking.

In some embodiments, the method further involves post presenting theuser state score and the performance score synchronized with the contentand the VR event to assist the user to better attain the desired userstates and desired manual inputs on the input device.

In some embodiments, the method further involves revising the content inresponse to the feedback provided to the user where the user is furthertrained on the revised content.

In some embodiments, the user state score further comprises failurebrain states.

In some embodiments, the display is a stereoscopic display.

In some embodiments, the method further involves presenting the contentand the user states and manual inputs of the user in real time on asecond display.

In some embodiments, the method further involves providing real timefeedback to the user during the presentation of the content.

Referring to FIG. 1 in accordance with an exemplary implementation ofembodiments described herein, there is provided a perspective view of atraining system 100 with a wearable device 105 and other computingdevice 160. The wearable device 105 has a stereoscopic display 110;bio-signal sensors 120; facial bio-signal sensors 130; sound generator140; a computing device 150; tracker 145; and user manual inputs such asmouse, joystick, or keyboard (not shown). The other computing device 160and the computing device 150 may be, for example, a computer, embeddedcomputer, server, laptop, tablet, or mobile phone. The stereoscopicdisplay 110 is a 3-dimensional (3D) (or dual 2-dimensional (2D) imagesgiving “3D”) display, but alternatively may be a 2-dimensional (2D)display.

The other computing device 160 is in communication with the wearabledevice 105 and provides the wearable device 105 with content to create aVR (Virtual Reality) environment. The VR environment includes aninteractive VR environment where content presented on the display mayupdate or modify in response to input from bio-signal sensors 130, othersensors, user manual inputs, or other inputs, for example. The othercomputing device 160 may also be a server over the Internet or othernetwork. In other embodiments, the functions of the other computingdevice 160 are incorporated into the computing device 150. In furtherembodiments, the functions of the computing device 150 are incorporatedinto the other computing device 160.

The tracker 145 is an inertial sensor for measuring movement of thewearable device 105. It detects the 3-dimensional coordinates of thewearable device 105 and accordingly its user's location, orientation ormovement in the VR environment including the user's gaze direction. Thetracker 145, for example, comprises one or more accelerometers and/orgyroscopes. The sound generator 140, for example, comprises one or morespeakers, microphones, and/or head phones.

In various implementations, the wearable device 105 may include avariety of other sensors, input devices, and output devices. Forexample, the wearable device 105 may comprise touch sensor for receivingtouch input from the user and tactile device for providing vibrationaland force feedback to the user. The training system 100 may furtherinclude input devices such as mouse, keyboard and joystick.

The wearable device 105 is, for example, a wearable headset worn on auser's head. The computing device 150 of the wearable device 105 isconfigured to create a VR environment on the stereoscopic display 110and sound generator 140 for presentation to a user; receive bio-signaldata of the user from the bio-signal sensors 120, at least one of thebio-signal sensors 120 comprising a brainwave sensor, and the receivedbio-signal data comprising at least brainwave data of the user; anddetermine brain state response elicited by the VR environment at leastpartly by determining a correspondence between the brainwave data and apredefined bio-signal measurement stored in a user profile, thepredefined bio-signal measurement associated with predefined brain stateresponse type. The brain state response may comprise an emotionalresponse type. The wearable device 105 may be called a virtual realityheadset.

The VR (stereoscopic) display 110 is positioned for viewing through aneye mask 125 wearable by a user. The eye mask 125 comprises an aperturefor each eye, and a plurality of facial bio-signal sensors 130positioned on the mask for contacting the user's face when the wearabledevice 105 is worn. One or more straps, which may optionally beadjustable, are attached to the eye mask 125 or display portion of thewearable device 105. Optionally, bio-signal sensors may be positionedalong one or more of the straps to sense brainwave activity in the userthrough the user's head. Sensors positioned along the straps may bespecifically configured to travel a distance from the strap, past theuser's hair, if any, to the user's scalp. Accordingly, any such sensorsmay include an elongated contact area, which is optionally of aresilient construction. The facial bio-signal sensors 130 measure,include, electrical bio-signals such as EEG, EMG (Electromyograph) andEOG (Electrooculography), as well as FNIRS (functional near infraredspectroscopy). The materials used to support the device on the face maybe opaque to the wavelengths used by the FNIRS sensors such that ambientlight can be reduced and thus increase the signal to noise ratio for thesensor.

Electrical signals may be measured on other regions of the head and maybe mounted to the supporting architecture of the display device.Typically these are elasticized fabric. Sensors that measure scalppotentials would typically have a fingered design to allow theconductive electrodes to reach through the hair to reach the surface ofthe scalp. The fingers should be springy to allow for comfort and allowfor the user to manipulate them in a fashion that will spread anddisperse hair to facilitate a low impedance interface to skin of thescalp. Capacitive electrodes may also be used. These would allow for aslight air between the electrode and the scalp. Many electrodes shouldbe used if possible to allow for a higher dimensional bio-signal tofacilitate denoising signal processing and to acquire more accuratespatial information of the bio-signal activity. Good spatial resolutionwill allow more precise interpretation of the electrical activity in thebrain as well as muscular activity in the face and head—which are vitalfor accurate emotion estimation. Facial bio-signal sensors 130 furtheryield facial expression information (which is difficult to obtain usingcameras in a VR headset.). Muscles specifically around the eyes play animportant role in conveying emotional state. Smiles for example ifaccompanied by engagement of the muscles at the corners of the eyes areinterpreted as true smiles, as opposed to those that are put onvoluntarily. EOG signals give information about eye movements. Basicgaze direction and dynamic movement can be estimated in real-time andcan thus be used as a substitute for optical methods of eye tracking inmany applications. Measurement of the EOG signal is also important fornoise free interpretation of the EEG signal. FNIRS sensors if used canprovide supplemental information about activity in the frontal region ofthe brain with high spatial accuracy. Other sensors tracking other typesof eye movement may also be employed.

Embodiments of the training system 100 may provide for the collection,analysis, and association of particular bio-signal and non-bio-signaldata with specific brain states for both individual users and usergroups. The collected data, analyzed data or functionality of thesystems and methods may be shared with others, such as third partyapplications and other users. Connections between any of the computingdevices, internal sensors (contained within the wearable device),external sensors (contained outside the wearable device), usereffectors, and any servers may be encrypted. Collected and analyzed datamay be used to build a user profile that is specific to a user. The userprofile data may be analyzed, such as by machine learning algorithms,either individually or in the aggregate to function as a brain computerinterface (BCI), or to improve the algorithms used in the analysis.Optionally, the data, analyzed results, and functionality associatedwith the system can be shared with third party applications and otherorganizations through an API. One or more user effectors may also beprovided at the wearable device or other local computing device forproviding feedback to the user, for example, to vibrate or provide someaudio or visual indication to assist the user in achieving a particularmental state, such as a meditative state.

In another aspect of embodiments described herein, the wearable device105 may be in a form of one or more sensors adapted to being placed ator adhered to the user's head or face. Each sensor may optionallycommunicate with one another either through wires or wirelessly. Eachsensor may optionally communicate with the other computing device 160either through wires or wirelessly. The other computing device 160 maybe mounted to the wearable device 105 in order to reside at or near theuser's head or face. Alternatively, the other computing device 160 maybe located elsewhere on the user's body, such as in a bag or pocket ofthe user's clothing or on a band or strap attachable to the user's body.The other computing device 160 may also be disposed somewhere outsidethe user's body. For example, the sensors may monitor the user, storingdata in local storage mounted to the wearable device 105, and oncemoving into proximity with the other computing device 160, the sensors,or a transmitter of the wearable device 105 may transmit stored data tothe other computing device 160 for processing. In this implementation,the wearable device 105 may be predominantly usable by the user whenlocated nearby the other computing device 160.

Optionally, the wearable device 105 may be used to implement aspects ofthe systems and methods described in PCT Patent Application No.PCT/CA2014/000256, filed Mar. 17, 2014 the entirety of which isincorporated by reference herein. Accordingly, the wearable device 105may implement a method that may involve acquiring bio-signal measurementfrom a user using the bio-signal measuring sensor during a VR event. Thebio-signal measurement may include brainwave state measurement. Thewearable device 105 may process the bio-signal measurement, including atleast the brainwave state measurement, in accordance with a profileassociated with the user. The wearable device may determine acorrespondence between the processed bio-signal measurement andpredefined device control action, which may also generate effects in theVR environment. In accordance with the correspondence determination, thewearable device may control operation of component of the wearabledevice or effects in the VR environment. Various types of bio-signals,including brainwaves, may be measured and used to control the device orthe VR environment in various ways. The controlling operation ofcomponent of the wearable device may comprise sharing the processedbrainwave state measurement with computing device over a communicationsnetwork. Thresholds of brain state may be learned from each user.

Optionally, the wearable device 105 may be used to implement aspects ofthe systems and methods described in PCT Patent Application No.PCT/CA2014/000004, filed Jan. 6, 2014 the entirety of which isincorporated by reference herein. Accordingly, the wearable device 105may be used with a computer system or method for guiding one or moreusers through a brain state guidance exercise or routine, such as ameditation exercise. This guidance may be referred to as training. Thesystem may execute a brain state guidance routine with a brain stateguidance objective; present brain state guidance indication at thecomputing device for presentation to user, in accordance with theexecuted brain state guidance routine; receive bio-signal data of theuser from the bio-signal sensor during a VR event, of the bio-signalsensor comprising brainwave sensor, and the received bio-signal datacomprising at least brainwave data of the user; measure performance ofthe user relative to brain state guidance objective corresponding to thebrain state guidance routine at least partly by analyzing the receivedbio-signal data; and update the presented brain state guidanceindication based at least partly on the measured performance. Thepresented brain state guidance indication may be feedback that provideseffects in the VR environment. The system may recognize, score, andreward states of meditation, thereby optionally gamifying the experiencefor the user to provide further training. The system, using bio-signaldata measurements measured by the wearable device 105, and in particularbrainwave state measurements, may change the state of what is displayedon the display of the wearable device. For example, in response to adetermination that the user has achieved a particular brain state, ormaintained a particular brain state for a period of time, the wearabledevice may update the display to provide an indication of thedetermination (e.g. indicating to the user what brain state has beenachieved, and, optionally for how long) and may further display anindication of a particular reward assigned to the user in response tothe determination.

Optionally, the wearable device 105 may be used to implement aspects ofthe systems and methods described in PCT Patent Application No.PCT/CA2013/001009, filed Dec. 4, 2013, the entirety of which isincorporated by reference herein. Accordingly, the wearable device 105may be used with a computer system or method for modulating contentbased on a person's brainwave data, obtained by the sensors of thewearable apparatus 105 during a VR event 430, including modifyingpresentation of digital content at a computing device or within the VRenvironment. The content may also be modulated based on a set of rulesmaintained by or accessible to the computer system to provide effectswithin the VR environment. The content may also be modulated based onuser input, including through receipt of a presentation control commandthat may be processed by the computer system of the embodimentsdescribed to modify presentation of content. The modification mayprovide effects in the VR environment. Content may also be shared withassociated brain state information to provide additional effects in theVR environment.

Optionally, the wearable device 105 may be used to implement aspects ofthe systems and methods described in PCT Patent Application No.PCT/CA2013/000785, filed Sep. 16, 2013, the entirety of which isincorporated by reference herein. Accordingly, the wearable device 105may be used with a computer network implemented system for improving theoperation of one or more biofeedback computer systems. The system mayinclude an intelligent bio-signal processing system that is operable to:capture bio-signal data and in addition optionally non-bio-signal dataduring a VR event 430; and analyze the bio-signal data andnon-bio-signal data, if any, so as to: extract one or more featuresrelated to individual interacting with the biofeedback computer system;classify the individual based on the features by establishing one ormore brainwave interaction profiles for the individual for improving theinteraction of the individual with the one or more biofeedback computersystems, and initiate the storage of the brain wave interaction (orbio-signal interaction) profiles to a database (for example, in acloud); and access one or more machine learning components or processesfor further improving the interaction of the individual with the one ormore biofeedback computer systems by updating automatically thebrainwave interaction profiles based on detecting one or more definedinteractions between the individual and the one or more of thebiofeedback computer systems.

Each person's brainwaves are different, therefore requiring slightlydifferent tunings for each user. Each person's brain may also learn overtime, requiring the system platform to change algorithm parameters overtime in order to continue to analyze the person's brainwaves. Newparameters may be calculated based on collected data, and may form partof a user's dynamic profile (which may be called bio-signal interactionprofile or user profile 335). This user profile 335 may be stored in thecloud, allowing each user to maintain a single profile across multiplecomputing devices. The user profile 335 may provide parameters used bythe user state estimator 325 to determine a user states during a VRevent which in turn may be used to compute feedback to the user whichmay have effects in the VR environment. Other features of the same oranother non-limiting exemplary implementation may include: improvingalgorithms through machine learning applied to collected data eitheron-board the client device or on the server; saving EEG data along withapplication state to allow a machine learning algorithm to optimize themethods that transform the user's brainwaves into usable controlsignals; sharing brainwave data with other applications on mobile devicethrough a cloud services web interface; sharing brainwave data withother applications running on client devices or other devices in thetrusted network to provide for the user's brainwave data to control oreffect other devices; integration of data from other devices andsynchronization of events with brainwave data aid in context awareanalysis (during the VR event to provide additional context, forexample) as well as storage and future analysis; performing time lockedstimulation and analysis to support stimulus entrainment event-relatedpotential (“ERP”) analysis; and data prioritization that maximizes theamount of useful information obtainable from an incomplete data download(i.e. data is transmitted in order of information salience).

The wearable device 105 may further be in communication with anothercomputing device, such as a laptop, tablet, or mobile phone such thatdata sensed by the headset through the sensors may be communicated tothe other computing device for processing at the computing device, or atone or more computer servers, or as input to the other computing deviceor to another computing device. The one or more computer servers mayinclude local, remote, cloud based or software as a service platform(SAAS) servers. Embodiments of the system may provide for thecollection, analysis, and association of particular bio-signal andnon-bio-signal data with specific mental states for both individualusers and user groups. The collected data, analyzed data orfunctionality of the systems and methods may be shared with others, suchas third party applications and other users. Connections between any ofthe computing devices, internal sensors (contained within the wearabledevice), external sensors (contained outside the wearable device), usereffectors (components used to trigger a user response), and any serversmay be encrypted. Collected and analyzed data may be used to build auser profile that is specific to a user. The user profile data may beanalyzed, such as by machine learning algorithms, either individually orin the aggregate to function as a brain computer interface, or toimprove the algorithms used in the analysis. Optionally, the data,analyzed results, and functionality associated with the system can beshared with third party applications and other organizations through anapplication programming interface (API). One or more user effectors mayalso be provided at the wearable device or other local computing devicefor providing feedback to the user, for example, to vibrate or providesome audio or visual indication in the VR environment to assist the userin achieving a particular mental state, such as a meditative state andprovide training to the user.

Sensors usable with the wearable device 105 may come in various shapesand be made of various materials. For example, the sensors may be madeof a conductive material, including a conductive composite like rubberor conductive metal. The sensors may also be made of metal plated orcoated materials such as stainless steel, silver-silver chloride, andother materials. The sensors include one or more bio-signal sensors 120,such as electroencephalogram (EEG) sensors, galvanometer sensors,electrocardiograph sensors, heart rate sensors, eye-tracking sensors,blood pressure sensors, pedometers, gyroscopes, and any other type ofsensor. The sensors may be of various types, including: electricalbio-signal sensor in electrical contact with the user's skin; capacitivebio-signal sensor in capacitive contact with the user's skin; blood flowsensor measuring properties of the user's blood flow; and wirelesscommunication sensor placed sub-dermally underneath the user's skin.Other sensor types may be possible.

In addition to or instead of processing bio-signal measurements on thewearable device 105, the wearable device 105 may communicate with one ormore computing devices (for example, the other computing device 160) inorder to distribute, enhance, or offload the processing of thebio-signal measurements taken or received by the wearable device. Inparticular, the one or more computing devices may maintain or haveaccess to one or more databases maintaining bio-signal processing data,instructions, algorithms, associations, or any other information whichmay be used or leveraged in the processing of the bio-signalmeasurements obtained by the wearable device. The computing devices mayinclude one or more client or server computers in communication with oneanother over a near-field, local, wireless, wired, or wide-area computernetwork, such as the Internet, and of the computers may be configured toreceive signals from sensors of the wearable device.

The bio-signal sensor(s) 120 may be provided by a separate wearabledevice, such as the device described in U.S. Patent Application No.61/924,020 or International Patent Application No. PCT/CA2015/000003,the entirety of each of which is hereby incorporated by reference,whereby the separate wearable device is in communication with thewearable device of the embodiments described herein, the VR device, ormay be used to implement aspects of the wearable device and integratedtherewith.

Referring to FIG. 2 in accordance with another implementation ofembodiments described herein, there is provided a perspective view of atraining system 200 with a display 225; Input devices 230; soundgenerator 240; a wearable device 215; and computing device 220. Thewearable device 215 has a headset with bio-signal sensors, facialbio-signal sensors and tracker modules with the functionality ofcorresponding to the bio-signal sensors 120, the facial bio-signalsensors 130, and tracker 145 of FIG. 1. The computing device 220 havingthe functionality of the other computing device 160 and the computingdevice 150. The display 225 having the functionality of the stereoscopicdisplay 110. The Input devices 230 and sound generator 240 similarlyhaving the functionality of the input devices and sound generator 140 ofFIG. 1.

The other implementation of FIG. 2 may function the same as theexemplary implementation of FIG. 1. An example difference between theimplementations is that the display 225 and the sound 240 are not partof the wearable device 215 whereas the display 110 and the sound 140were part of the wearable device 105. Any disclosure referring to FIG. 1also apply accordingly to FIG. 2 unless it does not logically apply. TheVR environment generated by the other implementation of FIG. 2 may notbe as complete as the VR environment generated by the exemplaryimplementation of FIG. 1, but may provide an adequate interactive VRenvironment to facilitate training.

Referring to FIG. 3 there is shown a system 300 view of an exampleimplementation of the training system 100 of FIG. 1 with a user 305wearing the wearable device 105. The system 300 may include varioushardware and software components such as biological signal acquisition340; biological signal processing pipeline 345; user state estimator325; data mining improver 330; context logic 310; manual inputs 320;user profile 335; and VR Application 315.

The user profile 335 contains the signal processing parameters tuned tothat user 305 and the parameters for the user's specific predictionmodels. The signal processing parameters are used by the BiologicalSignal Processing Pipeline 345 to interpret the user's bio-signal datareceived during a VR event. The parameters for the prediction models areused by the user state estimator 325 to predict a brain state (e.g.drowsy, or agitated etc.) or other user states such as eye position,high muscle tension etc. during the VR event. The user profile 335 mayoptionally store the history of all of the user's sessions, rawbiological data, processed data, demographic data, etc.

The VR Application 315 uses the user's intention and user state,including brain state, to control the content of a multimedia VRenvironment presented to the user to provide an interactive environment.The VR Application 315 drives the display 110, sound 140, tactilefeedback, etc. to provide effects in the VR environment.

The Biological Signal Acquisition 340 acquires the bio-signal data,including brain state signals, from the bio-signal sensors 120 and 130,and such other sensors that attach to the user's body. The bio-signaldata, analog signals, are amplified with high signal to noise ratio.

The Biological Signal Processing Pipeline 345 includes three categoriesof components: signal processing, feature extraction, and featureselection. Each of the components uses the parameters of the user 305from the user profile 335 to customize processing for that user. Signalprocessing is extraction or transformation of the bio-signal data. Forexample, it processes the bio-signal data to remove unwanted noise andartifacts and/or apply a filter to focus on frequency bands of interest.Feature extraction extracts measurements, variables, and mathematicaltransformation of measurements of the signal processed bio-signal data.Feature selection selects the features of the bio-signal data thatoptimize the accuracy of the user state estimator 325 for predicting thebrain state of the user.

The user state estimator 325 predicts the user state of the user basedon a prediction model using a combination of the features of thebio-signal data, manual inputs of the user through manual inputs 320,context data from the context logic 310 (for example what content isbeing present by the VR Application 315), and the user profile from theuser profile 335. The user state of the user includes the brain state ofthe user and other states of the user such as head/eye gaze direction.Where the prediction was incorrect, for example, as per user manualinput, the data mining improver 330 retrains the prediction model oradapts it based on manual inputs from the user. The data mining improver330 may also initialize the prediction model for a new user that has noprevious history with the system based on aggregate of other users'data.

The data mining improver 330 further may use previous VR sessions to addnew labelled data which is used to create a new prediction model oradapt an existing prediction model in the user state estimator 325. Theparameters of the new prediction model are stored in the user profile335 of that user.

The context logic 310 analyzes and determines what the user is doingusing manual input 320 of the user and user state estimator 345 (exampledirection of eye gaze). Also the context of the VR environment that theuser is in is supplied by the VR application 315. The context is used asan additional feature and or label of the biological features, forexample a truck suddenly appears, used by the data mining improver 330to update the prediction model in the user state estimator 325. Thedirection, location or orientation of the user 305 or other data fromtracker 145 or sensors 120, 130 may also provide input to context logic310 to determine context of the VR environment, such as, for example,the position of the truck relative to the user 305 within the VRenvironment or the orientation or eye gaze of the user 305 relative tothe truck within the VR environment.

Referring to FIG. 4 there is shown a flow diagram of a process forimplementing aspects of FIG. 1, FIG. 2 and FIG. 3 where the trainingsystem 100 has the user 305 wearing the wearable device 105. The processmay involve various operations that may control devices and contentdisplay. The trainee or user 305 is, for example, driving a car in atraining VR environment. The user 305 enters manual input 410 such asthose from steering wheel, gas, and brakes; while the user's 305biological features 420, such as brain waves and eye gaze directiondata, are sensed by the sensors 120 and 130. The biological features 420and manual input 410 are used to determine user state 425. The userstate 425 and the desired user state 435 optionally may only includebrain states and not include the manual input 410. The manual input 410is further used to determine its effect in VR environment 415. Theeffect in the VR environment 415 is used by the VR application 315 tochange aspects of the VR environment accordingly. The VR environment isinteractive as it changes depending on manual input 410 and user state425. Other inputs may also cause effect in the VR environment to provideadditional interactivity.

The VR application 315 is presenting a stream of content to the user 305thereby creating the VR environment. In the VR environment, there are VRevents 430 such as: scenario (a) a truck in the oncoming traffic;scenario (b) the truck wavering in the oncoming traffic; and scenario(c) the truck in the oncoming traffic moving into lane of the user's 305car. For each of these example VR events 430; a score 450 (or grade) isgenerated based on a comparison of the user state 425 during the VRevent to desired user state 435, and the effect in VR environment 415(or manual input 410) to desired effect 440. Higher scores 450,indicating better user 305 performance, are generated when there is abetter match between the determined user state 425 and desired userstate 435, and the determined effect 415 and desired effect 440. VRevents may be events that occur within the VR environment such as thepresentation of specific content to the user within the VR environment,sounds provided by output devices, video displayed, tactile feedback,changes in the user's environment such as temperature, humidity, actionsby the user within the VR environment, and so on.

Where the user 305 reached a certain threshold 460 score 450, thistraining or VR environment ends, but optionally it may also continue forthe user 305 to improve. The user 305 is provided with feedback 470 onevents where the user 305 may improve such as the user 305 freezes inscenario (c) when the truck moves into the user's 305 car lane. Thefeedback 470 is further interpreted by revise training 480 used by theVR application 315 to emphasize or change VR events 430 to customize thetraining for the user 305 to improve the user's score 450. The user 305optionally repeats the training content from the VR Application 315until the user 305 passes the threshold score 460 and/or until the user305 is satisfied with his or her performance. The feedback 470 may beprovided to the user 305 on the display 110 or 225 after thepresentation of the content and may include a user state score andeffects score over the course of the each of the VR events i.e. postviewing of the streamed VR events synchronized with user manual inputsand user states.

The feedback 470 to user 305 optionally may be provided in real time ornear real time to the user 305 while the content is being streamed forthe VR environment. This real time feedback 470 is given when the userstate 425 (or the effect 415) deviates from the desired user state 435(or desired effect 440), for example, as a sound through the soundgenerator 140 and/or a visual indicator in the VR environment beingpresented to the user 305. The feedback may be provided in real time ornear real time during the VR event. The feedback may have additionaleffects in the VR environment to update or modify the stream of content.

For VR events 430, there are associated desired user states 435. Theuser state 425 and desired user states 435, for example, can be a label(e.g. drowsy), ordinal (e.g. high drowsy) or numerical (e.g. 0.78probability of being drowsy). For one VR event 430 over a period of time(specific VR event), there can be, for example, different user states425 over that period of time. Further, the user states 425 may be astatistical distribution of user states for a specific VR event. Thestatistical distribution may also have a threshold which for example canbe associated with pass or fail for the specific VR event 430. The userstate is compared to the desired distribution of brain state for that VRevent and a user state score is determined. In one implementation thedistribution may be a gaussian distribution and the score is the numberof standard deviations of user state from the mean of the distribution.A user state score is the output of the compare between the user state425 to the desired user state 435; the distribution can be, for example,a label ordinal (e.g. high) or numerical. The user state score is thebrain state score where the user state data is brain state data.

For VR events 430, there are associated desired effects 440. The effectin VR environment 415 is controlled by the manual input 410 for scoringpurposes, but different manual inputs 410 may create the same effect inVR environment 415 (for example, turning the car a later time but at agreater steering angle). The effect in VR environment 415 and desiredeffects 440, for example, can be a label (e.g. slowing), ordinal (e.g.low speed) or numerical (e.g. 0.46 probability of slowing down). For oneVR event 430 over a period of time (specific VR event), there can be,for example, different effect in VR environment 415 over that period oftime. Further, the desired effects 440 and the effect in VR environment415 may be a statistical distribution of effects for a specific VRevent. The statistical distribution may also have a threshold which forexample can be associated with pass or fail for the specific VR event430. The effects are compared to the distribution of effects for that VRevent and a performance score is determined. In one implementation thedistribution may be a gaussian distribution and the score is the numberof standard deviations of effect in VR environment 415 from the mean ofthe distribution of desired effect. A performance score is the output ofthe compare of the effect 415 to the desired effect 440; thedistribution can be, for example, a label ordinal (e.g. high) ornumerical.

The score 450 is a combination of user state score and performance scoreto provide a combined, for example, numerical score and constructivefeedback to the user specific to the content of the VR Application forwhich he or she is being trained.

VR events 430 are associated with biological signals 340 because bothshare common timestamps through the context logic 310. This allowsanalysis of the biological signal 340 relative to a VR event 430. In oneexample, the VR event 430 can be a label for the biological signalstream 340 which can be used by the data mining improver 330 to createprediction models for the user state estimator 325. The time of a VRevent 430 can be associated with an interval of time within a biologicalsignal which can be labelled for machine learning or for the data miningimprover 330. In one example, let us say that the time of a VR event 430occurred at t_(vr1). As an example, an interval in the stream ofbiological signals 340 is chosen prior to t_(vr1) and after t_(vr1),e.g. the interval can run from t_(vr1)−200 ms_(to) t_(vr1)+1000 ms. Thelength of the interval is chosen based on the expected physiologicalresponse of biological systems within the body and varies for differentsystems and stimulus/response physiology. Also the time before thestimulus is important to analyze as this provides a baseline tounderstand the impact of the VR event 430 on the user's user state. TheVR event 430 can be considered as a stimulus and the biological signalsare associated with the user's user state 425 or in other words theirresponse to the VR event 430. Event Related Potential (ERP) are signalsseen in a user's brain signals in response to a stimulus such as anexternal audiovisual event. EEG can be used to infer a user's brainstate to a VR event using event related synchronization ordesynchronization, event related potentials, asymmetry acrosshemispheres, coherence across a set of electrodes, and power in specificfrequency bands. EMG can be used to determine a user's level of musculartension, their movement in time. ECG can be used to infer the level of auser's arousal. These biological features calculated by the biologicalsignal processing pipeline 345 can be correlated to VR events 430 andare used by the user state estimator 325 along with other featuresextracted by 345 to estimate a user state 425. These features of thebiological signals provide a rich set of information for the data miningimprover 330 to increase the accuracy of the prediction models of userstate 425.

Referring to FIG. 5 there is shown exemplary implementations ofdetermine feedback 470 in Tables 5A and 5B. Table 5A is look up tablethat has three columns: the first column is the user state score, thesecond column is the performance score and the third column is thefeedback 470 given to the user and/or instructor associated with thosescores. As an example, if the user state score is low and theperformance score is low then the feedback is for the threshold 460 tonot pass the user 305 and, optionally, to send the user 305 back tore-do the contents of the VR application 315. If the user state score islow but the performance score is high then the feedback to the user isto take a revised 480 training where the content of the VR Application315 is revised to include, for example, more of the specific VR events430 for which the user 305 is weak within the interactive VRenvironment. In another example, if the user state score is low due tohigh anxiety then a specific VR event 430 may be included to providemeditation training. For mediation training, VR events 430 are set up toreward the user 305 for attaining a meditative state, and also giveerror signals to the user 305 if they are falling out of the desiredmeditative state. If the user state score is high but the performancescore is low then the user 305 can be given VR events 430 within theinteractive VR environment that can help him improve his technicalknowledge. If both the user state and performance score is high then thefeedback to the user 305 may be pass this VR event and the user can moveon to the next sequence of content in the VR Application 315 in atraining program. Table 5B is a look up table for determining user statescore and performance score. Other scoring techniques may be used andthis is an example illustrative example embodiment. This is an exampleof training a truck driver who is in a car (as part of the VRenvironment) travelling down the left lane of a highway. There may be alarge 18 wheeler truck ahead (an example VR event as part of the VRenvironment) that is the source of a number of VR events posingchallenges to the driver under the training.

The user states 425 and desired user states 435 may compromise a largenumber of different brain states. Some exemplary brain user states areability of operator to learn; ability of operator to correctly predictaccuracy of their decisions; ability for emotional regulation;concentration; focus; sensory clarity; equanimity; mental workload;sensitivity to relevant external data; insensitivity to irrelevantexternal data; prior emotional states before performing; and eventrelated potentials in response to stimulus.

The score 450 can further include, for example, the following threetypes of scores: 1) ability of operator to learn 2) ability of operatorto correctly predict accuracy of their decisions and 3) emotional stateleading to impaired thinking.

The ability of operator to learn is an innate ability of the operator.High levels of frontal alpha (an EEG signal) are associated withincreased learning performance. This can be used to determine if theoperator is qualified for the position because there is an expectationof constant learning in these types of jobs.

The ability of operator to correctly predict accuracy of their decisionsis the difference in the operator's expectation of his/her error and thecorrect decision is known as prediction error. Prediction error can bemeasured from the operator's EEG. So if an operator expected that he/shemade the correct decision but the system informed them that theirdecision was wrong would produce a large prediction error. As anillustrative example, there may be two systems of thinking: System 1 andSystem 2: System 1: Fast, automatic, frequent, emotional, stereotypic,subconscious; and System 2: Slow, effortful, infrequent, logical,calculating, conscious. System 2 thinking may be less prone to error.System 1 thinking may be based on gut reaction or hunches. System 1thinking may lead to unreliable decisions.

The ability for emotional regulation are desirable emotional states thatenhance critical thinking in a stressful situation. Undesirable statesoccur for example in what may be referred to as an emotional hijackingwhere decisions may be driven by primitive response mechanisms,” such aswhen a center in the limbic brain proclaims an emergency and indicatesits urgency to the rest of the brain. This may happen before theneo-cortex has an opportunity to glimpse or appreciate the emergency.

Referring to FIG. 6, there is shown a side view of another exemplaryapplication of embodiments described herein. The training system 100 maybe applied to training of an operator in a VR simulation (VR Application315) of a nuclear reactor control panel. The user may be referred to as“Bill” for this illustrative example. Bill is in charge of operating anuclear reactor. As part of his training, Bill enters a trainingsimulator (training system 100) that shows him how to use the controlsat the reactor. The training simulator works through the wearable device105. In the simulator, Bill plays through several scenarios 610 (contentof VR Application 315) to evaluate his stress levels. Essentially, Billis practising or training for the moment that something dangeroushappens to the reactor, because the sequence for shutting down ordrawing down the reactor is complicated. As Bill plays through multiplescenarios generated as VR events within the interactive VR environment,the system logs his Brain State responses (user states 425) during theVR events. Bill's employers and supervisors learn when he experiencesstress, because the system has logged his responses 620 as feedback. Thefeedback may also be provided to Bill within the VR environment in someexamples. With this information, Bill's supervisors can also talk overhis anxieties with him, and emphasize certain areas of training. Billbecomes a more effective and confident employee. The next time thereactor experiences an anomaly, Bill may not panic and may know how todeal with the problem effectively.

In other words, video clips (VR content) of high stress may behighlighted. Bill is scored on task performance (performance score) andemotional state (user state score) 630. Teachers can also see the VRcontent and user's reaction in real time or near real time on anotherdisplay. A video of Bill's simulation is tagged with emotionally chargedor stressful points in the simulation of the VR environment (e.g. VRevents) so they can also be reviewed off-line with the user.

The training system 100 with the user 305 wearing the wearable device105 may be used for a wide range of applications. Various applicationsof the system according to some embodiments are now described. Theseapplications are merely exemplary implementations. Any feature orfunctionality expressed or implied in the respective applications is notintended to limit the scope of the embodiments. Similarly, variouscombinations of the applications described herein may be possible.

Helping Children Prepare for an Operation

This scenario focuses on children's needs. In this user story, a childwho is scared of an upcoming procedure (such a trip to the dentist, aflu shot, allergy shots, or a surgical procedure) can be taken throughthe experience of the process as part of the interactive VR environmentin advance to ease fears prior to the operation itself. For example,Johnny is scared to get his open-heart surgery next month at a majorchildren's hospital. Johnny's heart surgeon suggests that he play a gamein the training system 100 as part of the interactive VR environment, sohe can prepare himself mentally for the surgery. This is a new part ofthe mental health care regimen for incoming patients. It can beadministered to patients who live at a distance from the hospital.Johnny is told that it's a virtual surgery “game.” Johnny (user 305)walks through the surgery theatre or covers his eyes with a display ofthe wearable device (as example VR environments) and learns what eachtool does, and why he needs the surgery through interactions with the VRenvironment. He knows the name of everything when he's finished, becausehe's answered special quizzes and tests about the material. He's playeda bunch of games in the space that introduce him to what's about tohappen. The game actually replicates the specific hospital environment,so that by the time he enters the hospital, he is more comfortable andfamiliar, already knows where the bathrooms are located, the environmentseems less foreign or threatening and knows how to ask for help fromnurses. Johnny's responses (i.e. user state and, in particular, brainstate) to these VR events 430 (e.g. games, interactions with tools andother aspects of the surgery within the VR environment) generateJohnny's score 450. The training system 100 can recognize which aspectof the procedure creates the most stress for Johnny through the score450 and provides feedback 470 as part of the VR environment and duringthe VR events, in some example embodiments. In some examples, the systemmay possibly revise the VR environment 480 for additional VR events 430that can help Johnny be more comfortable. When Johnny's score 450reaches a certain threshold 460, he may be better prepared for hisupcoming procedure.

Social Awareness Community

Many games and VR experiences rely on clumsy, inaccurate models ofemotion, and don't create clear channels of communication betweenplayers or users, much less between players and characters and the userstate. In this system, users experience a world where artificialcharacters or anonymous players (examples of content within the VRenvironment) respond to their emotions and thoughts in an honest way,based on data from the system of the present invention. Self-awarenessis important to the process of therapy, and a component of the feedbackprovided by the system according to some embodiments. For example, Walt(a user) is in treatment for an anger management issue. He is the typeof person who compartmentalizes his anger until it explodes, rather thanexpressing it honestly. His therapist suggests that he re-learn how toexpress his emotions in a VR environment of the training system 100,where characters of the VR environment can react to virtual expressionsof his brain state (as example VR events and feedback) and not the wayhe pretends to feel. Walt uses the training system 100 of the presentinvention on a regular basis to access this construct (VR environment).The construct may be a casino floor, museum exhibit, shopping centre,bar, public park, or other crowded space. A bunch of agents andcharacters surround Walt, engaging him in conversation or activity, suchas games on the casino floor or in a carnival midway as example VRevents. The environment is designed to test Walt's emotions both in asocial way and in a stress-based way—when Walt loses games (i.e. VRevents 430), he gets angry (i.e. user state 425). The characters respondto Walt's emotional and mental state based on information gathered bythe system of the present invention generates his score 450, which maybe example feedback during the VR event. While Walt expresses emotiondifferently in the real world, or hides his feelings, the characters inthis world know how he's really feeling based on his EEG information. AsWalt wanders and explores this world, the way these characters respondto him as feedback reflects his actual state (i.e. through revisetraining 480), which he can't hide. When he's sad or negative, they tryto cheer him up, or they share their own negative feelings. When he'sangry they are fearful of him, or react with hostility to match his own(i.e. lowing his score 450). The feedback 470 is used to revise 480 theVR events 430 which drive the characters and scenarios in the VRenvironment. When he's cool and calm as determined by his brain state,they come and chat with him, or try to draw him out into a more outgoingand gregarious interaction. When he's excited and positive, they comeand dance, or discuss ideas with him. The training system 100 providesfeedback 470 to Walt based on his score 450 so that he may then improvehis future scores. Optionally, some of the characters may be real usersacting through the VR application 315.

Interactive Sensitivity Game

Most sensitivity training in the corporate world relies on perceivedaffective cues, emotional responses, body language, and other signalsthat are already difficult for insensitive people to discern orunderstand. The training system 100 according to some embodiments of thedevice allows the opportunity to create a VR environment where userslearn and practise sensitivity by engaging with characters in the VRenvironment where they wear their “emotions on their sleeve” (visuallyrepresented in real time via colour-coding, auras, icons, logos,avatars, etc.) based on determined user or brain states during VR eventswithin the VR environment. Socially interacting in this world may helptrain people to look through the surface and see the inner world withinpeople based on feedback. The world is gamified, so that players arerewarded for their ability to spread calmness, ease, happiness, andother positive emotions through social interactivity. The user 305 isscored based on their responses to these VR events 430. For example,Francine (a user) goes into an online VR gaming environment with herhead-mounted display and EEG combo (wearable device 105). Francineinteracts with all kinds of characters in this world via in-game avatars(VR events 430). Internal EEG-based states are visually overlaid on theavatar so Francine can see the emotional and mental state of thecharacters in real time, and they can see hers. This is done byanimating facial expressions perhaps even exaggerating them in acartoon-like way with colour enhancement such as red scale for anger,and blue scale for sadness (i.e. revise training 480). By reviewing herscore 450 and the feedback 470 from these VR events, Francine may seewhere she can improve her sensitivity by improving her future scores450. In addition, Francine's training may be revised 480 based on herfeedback 470 such that new scenarios and characters are presented to heras content within the VR environment driven by VR events 430.

Combating Stage Fright

Public speaking can be very challenging for some people. There arecourses that specialize in teaching people how to speak in publicwithout fear, but they don't necessarily deal with the realities ofsocial anxiety disorder or other real health care concerns that keeppeople away from speaking up. This training system 100 may help usersdevelop public speaking and personal skills in a VR environment, whereinthe system helps the user deal with feelings of anxiety. It would bebeneficial to both people who have anxiety dealing with public speaking,and to the people who are trying to help them, such as employers,therapists, coaches, and friends or family.

For example, Parker (user 305) is the valedictorian of his graduatingclass, which means he has to give a speech to everyone attending hisgraduation ceremony. Unfortunately, he is extremely nervous aboutdelivering speeches and presentations in front of large groups inpublic. He's really scared that he will mess this up in front of hiswhole family and all his friends. After writing his speech, Parker usesthe training system 100 to build or engage a virtual practise space orVR environment where he can practise giving his speech. The spacematches the physical environment where he will be delivering the speech:the size of the space, lighting and weather conditions, and how manypeople will be in attendance, as well as ambient noise like coughing,sneezing, murmuring, or the sound of mobile devices. After building orentering this VR environment using the device, Parker can practisedelivering his speech. The VR application 315 tests Parker's ability tohandle the stress of giving his speech by responding to his brain state.The speech delivery may be one or more VR events and audience responsemay also be VR events. If Parker is relaxed, the environment throws hima curveball (VR events 430) and starts some commotion in the virtualaudience. Peter's reaction (as determined by his brain state) would bescored. If Parker is nervous (as determined by his brain state), the VRapplication 315 calms down the audience so he can practise gettingthrough the speech. Depending on how Parker handles the curveballs, i.e.VR events 430 based on his score 450, the virtual audience will have apositive or negative reaction—lots of applause, or not too much applausethat are driven by feedback 470 and revised training 480. In someexample embodiments, feedback may be provided using VR events that reactor are driven by the user state.

Customer Service Training

Many employee training modules rely on videos, role playing, or simplepencil-and-paper quizzes, or basic computer quizzes. This trainingsystem 100 would allow businesses to train employees to deal withdifficult customers using a VR environment, brain state monitoring, andfeedback rather than relying on self-reporting of emotional states andother traditional training techniques. Further, it would generatefeedback data on difficult situations that would help other employersand employees understand the challenges of dealing with the generalpublic.

For example, Dante (user 305) just got a job at a major conveniencestore chain. The chain prides itself on delivering the same quality ofservice at each of its stores. Part of his job training is learning howto deal with difficult customers. While wearing a wearable device 105,Dante plays out several scenarios of difficult customers as VR events.This includes a few different scenarios of mugging and shoplifting. Thesystem simulates different customer behaviour (VR events 430) dependingon the system's evaluation of Dante's stress, calm or levels offrustration level (for example user state 425 versus desired user state435). As he gives in to frustration, the customers get tougher (feedbackor VR events). When he's able to calm his mind and focus (desired brainstate), the customers grow easier to deal with and eventually leave(additional feedback and VR events). Dante is rewarded by pleasantcustomers (feedback or VR events) when the system detects he is calm andis greeted by difficult customers (feedback or VR events) if anundesirable brain state is detected. Over time, Dante learns that theway to deal with tough customers is to focus his mind and maintain asense of calm, and also thereby increasing his score 450. And if Danteis struggling based on his score 450 and feedback 470, then different VRevents 430 are revised 480 such as meditation practice to help Danteimprove his equanimity.

Virtual Therapy Space

Post-traumatic stress disorder (“PTSD”) is an increasing problem.Soldiers, law enforcement officers, survivors of domestic violence, andeven those who have been serially harassed by bullies and classmates canexperience it—even years after the initial trauma. But most therapiesfor patients with PTSD focus purely on cognitive behavioural therapy,without an immersive or sensual element. This training system 100 helpspatients with PTSD identify and learn how to deal with triggers (VRevents) in a safe VR environment. It is just immersive enough to createthe necessary feelings that need to be processed for healing, but thatimmersion happens in a space the patient actually chooses.

For example, Oliver (user 305) is experiencing PTSD after a tour withthe armed forces overseas. Oliver sometimes enters fugue states orexperiences rages or crying jags that he can't explain. He doesn't knowwhat triggers these events. As part of his therapy to deal with PTSD, heenters a VR environment that emulates the places where he experiencedtrauma as different VR events. He wears a wearable device 105 to accessthis VR environment and score his brain states during VR events. Whilewearing the device 105, the device can read his inputs and determine hisbrain state during the VR events. As his anxiety increases, the devicetakes note of the change in baseline brain state and generates a score450 with feedback 470 with a timestamp at the time of the VR event 430,so Oliver and his therapist can determine what VR events exactly madehim so anxious. Over time, Oliver identifies what triggers his emotionalresponses, whether it's a loud noise or flashing lights or even acertain colour. In the VR environment, Oliver is able to expose himselfto this stimuli (VR events) in a safe way, and learn how to deal withthe stimuli on his own (i.e. user state 425 versus desired user state435), without over-reacting. His score 450 and feedback 470 would aguide on his improvements with time.

Improve Golf Performance

Many coaches can tell an athlete to get in “the zone.” Fewer of thosecoaches have access to direct brain-state data about whether an athleteis actually in that zone. The system according to some embodiments usesbrain state data to help athletes and casual players perform better ornail a specific technique. For example, Geoff (a user) wants to improvehis golf swing. He enters a VR “golf coaching” environment—a golf courseis simulated using VR technology. The VR headset (wearable device 105)also contains EEG sensors. Geoff tees up and practices his swing in theVR environment. As he get ready to swing (a VR event), EEG data is beingcollected. This data is used to overlay mindfulness information, i.e. VRevent 430, or feedback, on the training scenario. For example, if Geoffis relaxed and focused, the image of the golf club will glow green; ifhe is distracted, it will glow red. The glow and color is an example ofreal time or near real time feedback during the VR event. Geoff can alsoenter into a “mini-mindfulness mode” to refocus himself before he takesa swing. Geoff then swings the club when he reaches the ideal focuslevel. The club is embedded with sensors so the VR training program,i.e. sequence of VR events, can determine if optimal swing positioning,etc. (manual input 320), is being achieved. After the swing (i.e. manualinput 410), data corresponding to Geoff's performance is displayed inhis VR environment—i.e. how far the ball traveled, effect of wind, etc.by effect in VR environment 415. The training system 100 presents Geoffwith his score 450 along with his user state 425 and desired user state435 as part of the feedback 470 showing both mentally and physically hemay improve his performance for next time. A training programimplemented by the wearable device may be a sequence of VR events wherethe sequence of VR events may dynamically change to provide feedback inresponse to a user's scored brain state.

Another Example Application

In accordance with another exemplary embodiment, more than one copy ofthe training system 100 of FIG. 1 may be interconnected together, viathe other computing device 160, to create a VR environment for more thanone user to provide virtual interaction between users. FIGS. 1 to 5 areaccordingly modified for this other exemplary embodiment. This otherexemplary embodiment is applied in some of the scenarios as anillustrative example.

Other Scenarios

FIG. 7 shows an example visual representation 700 of content and/orfeedback relating to 3D brain activity in a VR environment according tosome embodiments. This illustrates an example of what a visualrepresentation 700 of 3D brain activity may look like in a VRenvironment to provide a visualization of feedback to the user withinthe VR environment. Embodiments described herein may be used forapplications that utilize real-time 3D visualization of the brainactivity as feedback or indication of the user's brain state. The VRdisplay may show spatial activity in the brain and in the body (e.g.look around to see inside your mind and in your body in real-time). Thismay be useful for biofeedback and psychotherapy. Neuro-feedback may belocation-specific. This type of highly specific training can be hard fora user to engage with because it is so difficult to learn the relationbetween auditory stimulus and state of mind. VR visualization offeedback may help the user learn the connection as well as techniquesfor engaging target areas.

FIG. 8 illustrates an example of a user interaction with the trainingsystem 100 to provide a visual representation 804 of inside the user'smind using a VR environment rendered on the stereoscopic display. Theexample schematic illustrates a wearable device 802 with bio-signalsensors 808, 810, 812 that provides visual representation 804 or contentin a VR environment. The visual representation 804 may provide a VRrendered view of the user's brain with graphics showing brain activitydepicting determine brain states. The bio-signal sensors 808, 810, 812provide bio-signal data to a device or processor 814 implementingbio-signal processing operations as described herein to determine brainor user states, which are in turn used to determine a user state scoreand provide feedback as part of the visual representation 804 in the VRenvironment.

FIG. 9 shows an exemplary application for fostering an emotionalconnection in a VR environment. As an illustrative example, theapplication may involve two users 902, 904 each with a wearable device906, 908 with bio-signal sensors to provide bio-signal data to a device910 implementing a synchrony analysis process to synchronize and linkthe bio-signal data from the two users 902, 904. A real time or nearreal time 3D visualization of a mental and physical state (an examplevisual representation of feedback) can be extended to seeing insidesomeone else's body (e.g. use it for two person interactions), to seeinside someone else's brain as they see inside yours. A visualrepresentation 912 in the VR environment may illustrate a collage of thetwo user brain states (provided as output from device 910). The visualrepresentation 912 may be displayed as additional feedback and/orcontent in the VR environment. Analysis of cross-state such as neuralsynchrony may be provided as output from device 910 and displayed as thevisual representation 912 in the VR environment. This may be useful toincrease intimacy between users 902, 904. Each user 902, 904 may have adifferent field of view (FOV) of the visual representation 912 and auser 902, 904 may see the same or different content as the other user902, 904 in the VR environment. Users can work to synchronize with eachother based on the feedback display in the VR environment. It can beaugmented with other imagery that the user can manually effect tointensify feelings and facilitate new types of telecommunication.

FIG. 10 shows an exemplary application for representing emotionalavatars 1010, 1012 in a VR environment according to some embodiments.The visual representation may represent emotional avatars in a VRenvironment as different VR events or types of feedback. Each user mayhave a wearable device 1002, 1004 with bio-signal sensors to providebio-signal data to a device 1006, 1008 configured to generate a userstate score, as described herein. The device 1006, 1008 may implementdifferent operations such as EEG analysis, emotion estimation, emotionmapping, facial expression mapping, and facial expression analysis. Thedevice 1006, 1008 may include one or more processors configured toimplement the various operations. The device 1006, 1008 may serve one ormore wearable devices 1002, 1004. Adventures or activities with othersmay be more interesting if people can see how the other person isfeeling by way of a visual representation in the VR environment. Theusage of the facial sensors (an example of bio-signal sensors ofwearable device 1002, 1004) may allow for the mapping of a user'sexpression to the face of their avatar 1010, 1012 in a VR environment(e.g. to represent detected facial states with associated smiles,squints, winks, furrows, frowns, etc.). This can be augmented with brainsignals from wearable device 1002, 1004 to do emotion estimation bydevice 1008, 1006. This estimate can further augment a charactersappearance in the VR environment as an example of feedback. A positiveeffect could be rendered as warm colours. A negative effect could berendered as cool colours. When users are excited they could spoutrainbows. This may form the basis for emotional gaming. An objective inthis example application may be to evoke emotions in others as VR eventsin a virtual world. Emotional state estimation can also facilitatesingle player storytelling, when the story knows that you are scared forinstance and the environment could alter to intensify the feeling.Different users may have different FOV of the VR environment.

FIG. 11 shows an exemplary application for a therapeutic visualrepresentation 1108 of a VR pet in a VR environment according to someembodiments. A wearable device 1102 has bio-signal sensors to providebio-signal data for true smile detection operation 1104 and a affectestimator operation 1106 (implemented by the wearable device 1102 oranother device connected thereto). It also allows for single playeremotional gaming which might be optimized for therapeutic use. Forexample, a VR pet that reacts well to true moments of user happiness, orgames when success is linked to cultivating different emotional states.The visual representation 1108 may update based on the user state scoreto provide feedback to the user.

Application: Entertainment—Calming Environment Uses EEG toRespond/Navigate

Embodiments described herein may use improved controllers to navigatespace in some examples. The navigation within the VR environment may bean example of feedback based on the user state score. Using vision andfocus/attention in the wearable computing device 1202 could allowsomeone to orient and focus to move, and still keep the experiencemoving and responding even without intention, revealing new elements.This would create a new way of interacting with and controlling VRenvironments based on brainwaves to provide effects within the VRenvironment as content and/or feedback based on the user state score.For example, Bill enters a virtual world designed for exploration andexperience via a wearable computing device. Rather than using a handheldcontroller, he determines his velocity in 3D-space by hisfocus/surprise/noticing his environment. Instead of an easy feeling ofcontrol, the environment seems to take him around and respond to hisbrain states in real time as example feedback based on his scored brainstate. Bill gets bored of flying around the space, and as he losesattention and relaxes, aspects of the VR environment change, and hismotion and velocity drastically slow down. Bill gets interested by thischange, and notices an engaging aspect of the VR environment. His newlevel of attention and gaze orientation navigate him toward that elementof the VR environment (which may be an example VR event) automatically.

The system may permit a range of inputs that together provide greaterresponsiveness, effects in the VR environment, feedback or a combinationthereof, including brain state inputs and other inputs in combination,for example: 1) noticing something salient, 2) focus, 3) attention, 4)head orientation, and 5) eye gaze. In this example, these inputs areprocessed by the VR system, and the output may include 1) anenvironmental response, 2) the user's position in the simulated 3Dspace, and 3) the user's velocity. In this example, a participant mayhave a wearable device 1202 with bio-signal sensors and a display toprovide a VR environment with a participant view 1208. A researcher mayhave a device 1210 with a display to provide a researcher view 1204,1206 that may update or modify based on the user state score of theparticipant to provide feedback to the participant and/or researcher.

Application: Entertainment—Virtual Tourist

Embodiments described herein may be used to implement an immersivevirtual experience, based on brain-state data gathered in real time fromthe wearable computing device 105. Unlike current virtual realitysystems that rely on a command-control system that is frequentlyhand-held, this one responds interactively based on data generated bychanges in the user's brain. For example, Janice wants very much to goto Paris. She downloads an immersive experience map of the city for useon the system which may be provided as part of the VR environment.Janice enters her map of “virtual” Paris to relax. Depending on hermood, emotion, or cognitive load, different layers of information appearon her “tour” of the city as part of the VR environment. Janice wandersaround Paris and enjoys photorealistic sights and amazing sound quality.As she gets a little less engaged with the sensory experience over time,information about the things she's seeing is overlaid on the screen(history and physical attributes of buildings, cultural references,etc.) As she gets bored, typical Parisian events start to happen in thevirtual world (a street performer calls out, a couple kisses, etc.). Allin all, the experience allows Janice to experience Paris, which is areal place, but in a surreal, engaging, and responsive way.

The system permits a range of inputs: mood, emotion, cognitive load,focus, drowsiness, etc. In this example, the inputs are processed by theVR system, and output may include feedback as: VR Events in the virtualworld, Overlaid information in the virtual world.

Application: Data Gathering—Evaluating Audience Engagement with a 3DEnvironment

Embodiments described herein may be useful for someone creating a videogame or any kind of 3D world could have the experience validated usingbrain state data. The present system including a combination ofbio-signal sensors with a VR device may allow designers to gather aunique and novel set of data from the user's brain. This combination mayallow designers for VR games and experiences to develop experiences thatare even more personal and immersive. This is an added value for gamedesigners and experience designers working with VR environments. Forexample, a video game may be generated for a head-mounted display. Tovalidate the world, users may be tested using a brain-sensor and HMD incombination. Motor areas, emotional areas, focus, attention, vigilance,drowsiness are all used to evaluate the user's engagement with thevirtual world and their interest in the game. In addition otherbiological sensors such as: heart rate, muscle activity, respirationrate, temperature etc. can be used to determine user's level of arousal,stress and other physiological parameters that are correlated toemotional state. The system receives a full report on moments ofweakness and moments of strength of the application and uses that toimprove the environment. The system also learns that certain types ofmotions/controls in their world generate emetic responses and feelingsof nausea, or other general feelings of discomfort like neck pain,eyestrain, claustrophobia, headaches, ringing ears, etc. The processiteratively continues to improve the environment and motion within tooptimize for positive experiences. is the system may be able to releasea game that takes a variety of physical responses into account, so thatmore people can play.

Embodiments described herein may permit such inputs as: a user's motoractivity, emotional activity, changes in brain-state, focus, attention,vigilance, drowsiness, nausea, and other responses. The system of thepresent invention creates such outputs as: raw metrics and data of userinteraction, such as movement, changes in focus and attention, changesin wakefulness, and other data that contribute to a portrait of userengagement. This data could then be translated into reports,infographics, and other valuable internal information for creatingbetter experiences.

Application: Entertainment—Experience Sharing and ResponseTracking/Shared VR Platform/Photo Sharing for VR

Services allow users to share their gaming experiences, but not withadded value of real time emotional data. Nor can those servicesreplicate real-world experiences that are actually happening. In thissystem, users can record experiences and share them with a community,and receive the collective emotional response of viewers back via ashared VR platform similar to photo- and experience-sharing platforms.In effect, this creates a social media platform for the sharing ofvirtual and emotional data. For example, Birut is on an amazing trip inNew Zealand and wants to share the experience. He uses his specialpanoramic camera to record the experience of a walk along a beautifulcliff side. Birut uploads the content to a shared social virtual realityplatform. This platform allows his friends to experience VR environmentscreated based on his camera input data, and annotate those environmentswith emotional data gleaned from the system. Jim, another user withaccess to Birut's profile, is able to access the content and experienceit as a VR environment via the wearable computing device. As Jimexperiences the cliff side walk (VR event), his mental and emotionalresponses are recorded and scored via the system to provide feedback. Afew hundred people experience the cliff side walk, attracted to it byauto-generated notes (examples of feedback or VR events) that othershave found it moving, engaging, as detected by their emotional responsesvia. Users and Birut himself are both able to access the emotional andmental data and see how people are experiencing the cliff side walk. Themoment when a bird flies by (another VR event) creates an interestingreaction, where some people are very drawn to that bird, while otherssimply ignore it. Users can see this clearly in the feedback dataassociated with the experience. The data shows how a moment where Birutmishandles the camera for a second causes some anger in a group ofpeople. Some percentage of the people start to get bored halfway throughthe experience and end it, while other stay engaged and persist to theend. Users subscribe to Birut's channel where he is generating qualitycontent.

The system of present invention can collate data from multiple inputs,including the EEG sensors, a camera, a microphone, motion detection, andtemperature.

Embodiments described herein can also generate data from EEG input thatenhances virtual environments and makes them more attractive topotential viewers. This makes the VR environments and content thereofmore participatory and dynamic.

Application: Data Gathering—Discomfort Detection

Contemporary virtual reality technology can create a lot of discomfortfor some users. The ability to detect discomfort (i.e. nausea, anxiety,heart rate) while using a VR device, so as to put a stop to badexperiences in the VR environment.

For example, Anne is wearing a head-mounted VR rig. She begins toexperience nausea in the environment. Embodiments described herein useEEG sensors to determine that she is experiencing a change in herbaseline brain-state. The changes in brain-state that Anne ismanifesting match a profile the system has established for physicaldiscomfort. The system sends this data to the VR device. The VR devicesends Anne an output (feedback) that helps her check-in with herdiscomfort. In this scenario, that could be a simple text message in theheads-up display, or even a character in the VR environment who asks ifthe player or user is feeling well. If the discomfort persists, the VRdevice offers Anne a way of opting out or ending the experience quickly.

Embodiments described herein allows for multiple inputs from thewearable computing device, including EEG sensors, motion detection,heart rate, eye-tracking, and other data that can help the systemdetermine whether a user is feeling discomfort.

Embodiments described herein can create outputs like text messages, apause in the gaming or virtual experience, dimmed lighting, changes intransparency and opacity, or even characters who can respond in realtime to players/users.

Application: Data Gathering—A Room of One's Own

Designing a VR environment can be a special exercise that creates ameditative mindset. Akin to the “mind palace” idea of cognition,designing special places with an emotional meaning for the user can be avaluable tool for people using virtual reality technology. Re-creatingan environment that triggers major emotional changes in the user, suchas a childhood bedroom, or a special place in a person's history, oreven a museum exhibit.

For example, Virginia has to move to another country for work. She has avery high-stress job. In order to help herself remain productive atwork, Virginia creates a virtual meditative space. Virginia slowlybuilds this space in her VR device by selecting dimensions, colours, andtextures. Then she adds in features from photographs and advertisingmaterials. She can even shop online for elements to add to the room,like a fountain or sculpture. Embodiments described herein allowsVirginia to move elements of the room, or change their shape or colour,only when she's focused (desired brain state), which may be an exampleof feedback. This is applying focused attention to the principles ofdesign to enable movement of objects and creation of VR space asdifferent forms of feedback. Slowly, Virginia learns to focus by makingand re-making the room as a meditative exercise.

Embodiments described herein can input data from EEG, heart rate,eye-tracking, and other sensor data and translated it into VR maps withresponsive changes to colour, music, opacity, transparency, and otherelements that help create a more immersive virtual world.

Application: Entrainment—Preventing Boredom During Exercise

Many exercise programs have no way of controlling for when members getbored. This system provides constant stimulation to people duringexercise via a VR environment, so that they don't get bored or focus toomuch on exertion such that it distracts them from accomplishing a goal.This system is engineered to constantly maintain attention away fromexertion by providing stimuli or feedback based on changes inbrain-state, gathered from EEG sensors in the system.

For example, Liana is trying to exercise more. But she finds itdifficult to stick with any one exercise for a prolonged period. At thesame time, she doesn't enjoy aerobics or weightlifting classes. One ofthe trainers at the gym recommends that she use the system to maintainher attention while she is on machines like treadmills and bikes. Thewearable computing device provides her with visual stimuli as VR events,based on a suite of available content. She could be in a naturalenvironment, or an urban one, or even a fantasy environment like MiddleEarth or a space station. If Liana maintains her heart rate and focus(detected through by the system), the environment stays in place asfeedback. If she loses focus or her heart rate, the environment beginsto fade away as further feedback. In another embodiment, the greaterLiana's engagement gives her additional power to play a game in the VRenvironment. This takes Liana's mind off the exercise, as she learnsmore about her fantasy environment or plays a neuro-feedback based game.

Embodiments described herein can translate inputs like heart rate; EEG;eye-tracking; pulse; gyroscope; accelerometer into outputs like dynamicchanges to virtual environment based on changes to wearer's brain state,such as changes to music, lighting, tempo, sound effects, colour,activity.

Application: Entrainment—Developing Attentional Bandwidth

Contemporary methods of training students for tests can only offervariations on tests, without any data on how students are focusing. Thiswould help train students and others to develop greater focus andvigilance over prolonged periods of time, with the added benefit ofincorporating brain-state data into the mix. With brain-state data,students, parents, teachers, tutors, and counselors can actually have anaccurate picture of what helps students engage with the material theyare attempting to learn, and how much time they actually need to study.

For example, Dennis has a hard time focusing in class and at home, whendoing his homework. Dennis' parents download an app into his wearablecomputing device according to some embodiments to help him learn how tofocus. The application takes a gamified approach to developingattentional bandwidth. It rewards Dennis with extra points and badges asfeedback for completing tasks in a calm, focused frame of mind (desiredbrain state). Sometimes these tasks are timed, but not always. He canchoose the types of tasks he would like to perform in the virtualenvironment: catching butterflies, setting up dominoes, going sailing,etc. Over time, Dennis gains more attentional bandwidth. He is able tofocus more intently on the tasks he's been assigned.

Embodiments described herein can track data inputs related to focus andattention, such as EEG sensing, eye-tracking, heart rate, and motion,and translate it into outputs like dynamic changes to a virtualenvironment, text messages, virtual prompts, and reports on attention.

Application: Entrainment—Developing Attentional Bandwidth in theClassroom

The majority of current testing mechanisms in schools reflect only acertain way of taking tests which is growing increasingly vulnerable tocheating and does not reflect the way students actually learn. The VRsystem according to some embodiments provides schools and students withalternative testing mechanisms that may be more user-friendly, that alsoshare data on student brain state and performance. This data can then beused to bring up test scores, improve tests, and benefit students andtheir schools.

For example, the town of East Dillon has students with low test scoresacross the board. As part of accountability legislation to help studentsraise test scores, the school enters a pilot program that charts focusand attention among students during tests. The goal is to raise testscores by reducing test anxiety and helping students focus solely on thetest, and not on their classmates or the surrounding environment. Duringtests, students at East Dillon High School put on wearable computingdevices of the present invention, and perform their tests in a virtualenvironment. The test presents as a game (as a sequence of VR events),where students choose different answers from a multiple choice list byclicking, pointing, blinking, or otherwise interacting with the virtualenvironment. This VR environment has none of the usual distractions of aclassroom. It can be quieter and friendlier, with calming sounds and noclassmates. As students perform the tests in the virtual environment,the system charts their ability to calm down and focus on the test. Thesystem generates reports (feedback) for students on times when they feelthe most anxiety or distraction. This is matched against existingprogress reports, academic calendars, and other personal and healthdata. Students and parents can see how students are really feeling asthey perform during tests. These reports can be used to talk withcounselors, therapists, or academic advisors to help refine studytechniques. Having learned how to calm down during tests, students canfocus better on major tests like the PSAT and SAT, or during state-widetests.

Embodiments described herein can take inputs such as EEG data,eye-tracking, and motion-detection and translate it into outputs such asdata metrics and reports, virtual prompts, and real time charts ofstudent attention as different examples of feedback.

Application: Entrainment, Therapeutic—Learning how not to be Creepy

Learning how to behave in a romantic or social context is difficult.While many online sources promise to teach “pick-up artistry,” thesesystems are often geared toward gullible people and against the“targets” they want to prey upon. This system would allow the user toprototype romantic and other interactions in a shared VR environmentthat teaches people how to develop appropriate boundaries while alsoencouraging respect and humanity. This would also be useful to peoplewith all types of social anxiety disorder, to help them learn how tointeract with people in a healthy way.

For example, Morley has zero success with the women he wants to date. Hehas no idea what to do about it. When he asks his friends sincerely,they all agree that he has trouble understanding how to draw appropriateboundaries—what is respectful behaviour, which jokes are okay to make,and how close he should stand to people without invading their personalspace. His therapist suggests he log regular sessions with the systemaccording to some embodiments to prototype interactions with women, anddevelop appropriate boundaries. Morley enters a virtual environmentwhere he can practise speaking to women. His therapist also enters theenvironment with him, to see how he's doing. During his conversationswith women in the virtual environment (VR events), Morley receivesvisual cues (as feedback) like colour changes, changes in music, oricons and alerts that tell him how his overtures are being received bythe people he's talking to. These are based on accumulated aggregateprofiles from other players, and from baseline brain state data gleanedfrom other users who have elected to make their anonymized dataavailable to developers. A simple red/yellow/green card system may beused in some examples for Morley to learn more about how his behaviouris impacting the people around him, and whether or not they want him tocontinue behaving in that manner. When he notices yellow or red cards,he can ask for further assistance and find out what he's doing wrong. Inthe virtual environment, other people feel safer telling him what he'sdoing well and what he's doing badly. They can be more honest. Based onexisting user data gleaned from other wearers, he learns how tocommunicate with women more respectfully. Eventually, Morley developsbetter communication skills that help him in real life.

Embodiments described herein can translate inputs like EEG data,eye-tracking, pulse, temperature, and motion into outputs like datareports, dynamic chances to the virtual environment, visual cues,colourful auras, text messages, and chimes to alert the user to theirbehaviour as feedback.

Application: Virtual Telepresence Presentations Enhanced by EEG VR toShow Real-Time Shifts in Audience Emotions

Most leadership training experiences don't offer real time brain-statedata. This system does allow for exactly that. Speakers, Executives andothers within leadership positions can gain real time understanding ofaudience reactions to their speeches and presentations as feedback. Thisway, they can gauge the effectiveness of a given speech or presentation,and adjust on the fly, especially if they have made a mistake or saidthe wrong thing.

For example, Joan is a major executive who is delivering a presentationat the yearly developers' conference. She wants to know what effectshe's having on the audience in real time. Some, all, or a percentage ofthe audience are also wearing wearable computing devices of the presentinvention. Some of them are in the room with her, and the others are inother locations, either watching via the internet or in other places inthe conference centre. She uses a wearable computing device of thepresent invention to gauge audience reactions as she speaks. Seeing thereactions of a whole virtual audience through the VR goggles provides avery strong, realistic and instant form of feedback to the speech giverthat is tied very closely in time to the wording or intonation that mayhave caused a pleasant, or adverse reaction from the audience. When Joanmakes a joke, she sees the emotional shift in the audience as they paycloser attention. As Joan leads the audience through her speech, she'sable to see how they react to what she says via the wearable computingdevice. She witnesses audience reactions through her device, whichmanifest as colour changes or icons or popups or logos in her vision. Asshe learns more about the audience's reaction, she can edit out certainjokes, or speed up the presentation to keep audience attention. Usingthis information, Joan delivers a far more effective presentation.System inputs may include EEG, EMG, ECG, GSR linked to VR events. Systemoutputs may include visual/vibrational feedback cues.

Application: Entrainment, Therapeutic, Entertainment—Arousal LevelModulating Virtual Sexual Encounter

There are many methods of couples therapy available to the people whoare interested in them. Very few offer enhanced sexual awareness basedon real time changes in brain-state, or the enhanced sexual focus andgreater intimacy between partners when that data is made available. Thestrength of this system is that it makes plain all the things that canbe difficult to express during an intimate encounter, even if thatencounter is with a trusted individual. With brain-state data, suddenlyfeelings of discomfort or pleasure are easy to recognize, and consent isfar simpler to discern.

For example, Paul and Linda are in a sexual relationship. Both of themhave problems focusing on sex. One has ADD, and the other has stress atwork that makes it difficult to “turn off” thoughts about the job. Thismeans that both of them have a tough time being truly emotionally andphysically intimate with each other, when they do have sex. Their sextherapist prescribes sessions with the wearable computing deviceaccording to some embodiments described herein that will help them focuson each other during sexual encounters, as a means of tuning outunhelpful thoughts and bringing their attention back to each other.Either together or separately, both partners can wear a wearablecomputing device that gently focuses their minds while also subtlyarousing them. This could be purely visual stimuli like traditionalpornography, or hearing the sounds of sexual activity, or even listeningto a florid audiobook. As arousal increases (desired brain state), thesystem helps detect both levels of arousal and levels of focusedattention. As the sensors pick up greater focus, more arousing materialcomes through the device as feedback—images, sounds, etc. all arrive ina steady stream. When Paul or Linda is unfocused, the stream fades out.Eventually, Paul and Linda learn how to “tune out” unwanted stimuli andfocus only on what is arousing them. They incorporate the system routineinto the “cognitive foreplay” phase of their sexual encounters, as alead-up to physical foreplay and sexual activity.

System inputs may include EEG, EMG, ECG, Accelerometer, GSR. Systemoutputs may include auditory, visual, vibrational.

Application: Entertainment, Diagnostic/Monitoring—Virtual MovieStoryline that Modulates Based on Emotions

Many narrative platforms, from novels to comics to films, have offeredalternate endings or differing branches of events. Very few can tailorthat experience in a real time dynamic way, such as the system of thepresent invention can do. Further, this system offers an enhanced movieexperience; an opportunity to “choose your own adventure,” based on realtime emotional reactions.

For example, David is watching a movie at home via his wearablecomputing device 105 of the present invention. The device continuouslytakes readings from him about whether his attention is drifting orremaining focused. During times when the system reads a lack of focus,it sends a signal to the wearable computing device 105 to make thecontent more exciting: there's an abrupt change in the music, or thecolours become more saturated, or the contrasts sharpen. Over time, thewearable computing device shows David several different versions of thesame film, based on his responses. David can save these differentversions of the film and share certain moments from them with hisfriends via a social network.

Application: Entertainment, Diagnostic/Monitoring—Virtual Report thatModulates Based on Emotions

For example, Roscoe has to read multiple productivity reports, as partof his job. The reports are very boring, though, and Roscoe has a hardtime focusing on them. He inputs some of the reports into a wearablecomputing device 105 of the present invention, which allows him to readthe reports without distraction. Not only are the reports right in frontof him when he sits down to read them, the device helps detect his focusand distraction. When Roscoe experiences distraction, the device sensesthe change in baseline brain state and provides him with stimuli to keephim on task as feedback. This stimuli could be a simple chime or sound,or perhaps a colour change, or maybe just a quick reminder text. Thedevice records how long it takes Roscoe to read his way through adocument and gives him progress bars to help him understand how he'sreading and at what speed. The device also charts when he feelsdistraction based on the text or when he notices an error in the text,or when he registers surprise. It marks the text accordingly. Thesepoints in the text can become places for Roscoe to speak with thewriters of the reports about their work.

Application: Therapeutic—Out of Body User Therapy

Physical therapy is a chore. And for patients with difficulty accessingclinics, actually attending sessions can be quite prohibitive. But witha wearable computing device 105 of the present invention, patients withphysically debilitating injuries and illnesses can perform physicaltherapy in a fun environment, while their emotional responses andsensations of pain are monitored. This can remove some of the emotionalburden from physio and occupational therapists, some of whom havedifficulty judging when a patient should continue within a givenactivity. With data provided by the device, patients can train forlonger, and therapists can see how they are actually performing withoutconfirmation or other biases getting in the way.

For example, Liza has experienced a catastrophic automobile accident,and needs intense physical therapy to repair damage to her shoulder.However, she has a hard time focusing on her exercises, and finds themboth boring and embarrassing, because she has such a hard time withthem. Her physiotherapist advises that she perform her exercises in avirtual environment where she can focus on accomplishing the tasks. Thephysiotherapist also advises her to use EEG sensors to monitor heremotional responses to the exercises and her sensations of pain. Lizawears the device and performs a variety of activities in the virtualenvironment that stretch and use her injured shoulder. For example, shemay play tennis or learn how to hunt with a bow and arrow. As sheperforms these activities, she not only uses the injured shoulder andhelps to heal it, but the device detects her sense of having made errors(ERNs), and her ability to focus on the exercises. Over time, Lizalearns how to focus appropriately on her exercises. She also feels lessstress and anxiety while using her injured shoulder.

System inputs may include EEG; gyroscope; accelerometer; moisturedetection; temperature. System outputs may include dynamic changes tovirtual environment based on brain state; data stream; progress reportsas feedback.

Application: Data Gathering—how do Different Architectures or InteriorDesigns Make Me Feel?

Many design companies rely on extensive prototyping to help decide whatthe best design for a given object is. But with the wearable computingdevice 105, in some example embodiments, they can gather real time brainstate data about how objects and affordances are being used by theintended audience, or research group. This system provides a valuablecontribution to user productivity based on response to ergonomics,affordances, and designs based on disability.

For example, there may be an application for developing a new interiordesign philosophy for a new office building. Before going through thecostly process of constructing the actual interior, the system canengage in user testing of a variety of environments. Office workers maywear VR headsets with EEG-reading technology built into the headset. Thetest subjects are exposed to a variety of ergonomic options for the newoffice space, i.e. open-concept vs. closed offices; various colourcombinations; various seating options; lighting levels. Employees engagein simulated work tasks while their ergonomic environments are adjusted.The EEG-reading device in the VR headset captures their moods andthought processes during each iteration of change that an applicationwishes to test. The brainwave responses of the test subjects arerecorded in relation to whether each change produces a positive ornegative result in terms of metrics that benefit productivity. After thetesting period, the system can suggest an ergonomic scheme thatmaximizes a productivity response.

System inputs may include EEG; gyroscope; accelerometer. System outputsmay include data.

An example of this application may be seen in FIG. 12 which shows a userinterface for architectural design or game design. This drawing showsthe participants view of an architecture. The researcher view providesthe user's field of view in real time during the user assessment. Theresearcher view after shows a colour coded heat map of the variousbiological states the user was in when in a certain part of thearchitecture which may be an example of feedback. This may allow theresearcher to redesign the architecture based on map of environment ofuser's brain states. The system can aggregate multiple participants inthis way to generate analysis map.

Another example may be seen in FIG. 13 showing a visual representation1300 of the VR environment of a user who is working and who desires tofocus on what he is doing. A bio-signal enabled headset can be used suchthat the rendering engine diminishes their field of view when they arenot focusing to limit sensory input when they are trying to re-cultivatetheir attention. The system may limit the FOV by darkening edges, or tochange the information density (polygon level changed). Saturation maybe modified. Detail may be made to disappear. The rendering may be doneso that changes in user state are not too intense so that user doesn'tget information shock. A Converse operation can also useful forachieving the same goal: as the user focuses, the amount of surroundinginformation decreased so that the central information of interest isintensified. This allows more information to be incorporated when theuser wants it. If the user focuses on something then more detail willappear. VR effect can also include zooming. Ken burns like effect forVR. When the user is focusing, the region of interest becomes magnifiedas feedback or effect (in a 3D word this can be done with greater effectthan 2D tricks of focus) and with extra details as shown in FIG. 14. Auser may have a wearable device 1402 with bio-signal sensors to providebio-signal data to for bio-signal measurements 1404, focus estimator1406, and detail rendering 1408 operations (implemented by wearabledevice 1402 or another device). The bio-signal measurements 1404, focusestimator 1406, and detail rendering 1408 operations may determine auser state score to provide feedback as a visual representation 1410 ofextra detail in an area of interest. A desired user state may be basedon a level or threshold of focus as determined based on the receivedbio-signal data from wearable device 1402.

Application: Therapeutic, Entertainment—Transport Entertainment System

In-flight entertainment is usually screen- or paper-based. This systemprovides stimulating entertainment and distraction to people withintense phobias during times of extreme stress, such as during aturbulent flight. There's a double-level of stimulation that helpspassengers calm down using the system of the present invention, whilealso distracting them with VR technology. Both of these elements work inconjunction to help passengers feel safer during their flight. Forexample, Billy is afraid of flying, but he has to do it regularly forwork. In order to get over his fear of flying (or at least make theflights more endurable), his therapist suggests to him that he purchasea wearable computing device according to some embodiments to distracthim during flights. During his next flight, the aircraft is experiencingturbulence. The device detects Billy's stress through EEG, and detectshow bad the turbulence is through an accelerometer and gyroscope thatjudges how much he is moving around. The VR system counteracts theturbulence by immersing him in a different world. He wears a VR headsetwith EEG sensors baked into the hardware that throws stimuli at thevirtual environment until Billy is suitably distracted and calm. Billy'svirtual environment is constantly changing to reflect the time of day,etc. His EEG sensors detect his level of alertness/drowsiness. If Billyindicates to the system that he wished to stay awake, the virtualenvironment will assist him by providing an alert if he is gettingdrowsy. Similarly, if Billy indicates to the system that he wishes tofall asleep, the VR environment will change to be conducive to rest.Billy can program the system in “sleep mode” to awaken him at a certaintime. Important announcements from the captain/train driver, etc. can befed directly into the VR system and given priority so Billy can respondto them immediately. If Billy suffers from a fear of flying, the VRsystem will compensate by activating a meditation program or byfiltering out distressing noises/visual stimuli.

Application: Therapeutic, Entertainment—Using Biometric Inputs to ModifyAnxiety/Stress Levels Based on Desired User Environment

This system adds a stress test component to in-game activities that areanother way of playing the game or judging progress. For example, Dan isan avid gamer. He likes high-action shooters like Call of Duty or Halo.Dan plays in a virtual reality milieu using a VR headset that containsEEG-reading sensors. Dan wants to increase his performance in thesegames so that he can perform well under high levels of stress. The VRgame presents a series of tests that put Dan under simulated stressevents—i.e. a sniper attack, etc. The EEG sensors in Dan's VR headsetasses his emotional/stress state during these tests and provide him withupdates based on his performance. Dan's performance during these “stresstests” can unlock different achievements and new missions. If Dan has astressful job—i.e. in the military—these “stress tests” can provide avaluable metric for gauging how he will perform in real-life situationsthat are stressful.

Application: Entrainment—Deliver Educational Content Effectively toStudents (Making Learning Fun)

This system gives students another way of learning information in adistraction-free environment where they can focus on their studies andreceive feedback from their brain state. For example, Sally is astudent. She learns in a classroom that uses VR equipment. She wears aVR-enabled headset during classes. Lessons are delivered with VRtechnology: i.e., she is in the middle of a virtual solar systemoriented around her. EEG sensors in the headset monitor Sally's levelsof attention and engagement with the subject matter are determined andscored against desired states. When Sally loses her focus, the systemalerts her teacher as feedback; it also provides her with extra stimulias additional feedback so that she can re-focus more naturally. The rateat which content can be delivered to Sally is modified automatically bythe VR headset she wears. Eventually, Sally's in-class and academicperformance improves, as she learns to focus on the tasks at hand.

Application: Therapeutic—Couple Counseling Using VR Technologies toEnhance Therapeutic Outcomes

Many systems of couples therapy focus solely on cognitive behaviourtherapy, or “talk therapy,” wherein each partner takes turns trying tomake themselves clear. This system allows couples and other partners tomindfully focus on their relationship with real time data streams fromeach partner, so they eventually learn how to “put their heads together”to solve problems and co-exist more peacefully. For example, Ted andCarol are a married couple who are receiving relationship counselingusing virtual reality-enabled technologies. As their counseling sessionsprogress, EEG-detecting sensors in their VR goggles monitor their brainstates. During the course of therapy, the therapist can instruct Billand Carol to focus on their mindfulness; the EEG-sensing technology candetect when their mindfulness state is suboptimal through scoring. Thetherapist can direct Bill and Carol to focus on their breathing or othertechniques to enhance mindfulness and relaxation. Bill and Carol canparticipate in shared trust exercises that are mediated through theircommon virtual environment. For example, they can perform trust falls,or defeat a common enemy together in a virtual reality game. Thisprocess encourages shared empathy between the couple. If one partner isdisturbed, the other sees this as a direct visual manifestation, such asan aura or an icon that appears within the virtual environment and sayswhat the other person has difficulty saying.

The system can translate EEG input along with motion-sensing,eye-tracking, temperature and pulse into dynamic visual feedbackinformation in an immersive virtual environment, almost like an “alarmbell” for when a conversation has gone bad or broken down between twopeople.

Application: Therapeutic, Entrainment—Sensitivity Training in a VREnvironment

This system allows corporate executives and others in leadershippositions to see themselves in the third person (as an example offeedback) as they are behaving with their employees.

Jorge works for a large, multinational corporation. As part ofdisciplinary training, Jorge is compelled to undergo a sensitivitytraining seminar that uses VR technology. The idea is to expose Jorge toa variety of situations in which he does not enjoy the privilege of hisdominant position in the social hierarchy. When the seminar begins,Jorge is invited to put on a pair of VR-enabled goggles that containEEG-sensing equipment. The other members of the sensitivity traininggroup are also instructed to do the same. The training program begins:Jorge sees himself—i.e., his avatar in the constructed VRenvironment—from an external position, so it is like Jorge is witnessinghimself and his actions in the third person. As the sensitivity trainingseminar proceeds, Jorge views himself under changing situations, i.e.:he sees himself as part of a different racial/ethnic group; as part of agroup with a different ability, status, etc. The other people in thegroup observe Jorge's avatar in his new states as they cycle through, asper the instructor's wishes. Both the EEG signatures of the group and ofJorge are recorded during the testing period. Jorge can watch as the EEGpatterns of the other people in the therapy group are “played back”during the course of the therapy session; they are played back andinterpreted visually much in the same way as a song is played back andvisually interpreted as part of a “visualization program” for musictracks. In this way, Joe can see how his transformations from physicalstate to physical state are “interpreted” by others in his group asactual EEG brainwave data. Joe reports how this experience haschanged/benefited him as part of a post-group post-mortem. Theinstructor can therefore gauge the effectiveness of the class based onhow Jorge interprets the EEG patterns of the other participants.

Application: Therapeutic—Meditation Simulation for Environments notConducive to Restful Contemplation

While meditation can be felt, it's hard to quantify in a way that otherpeople can understand. It's also easy for people doing meditation tofeel as though they've slipped behind or aren't making enough progress.This creates a distraction that is anathema to the act of meditationitself. Embodiments described herein may allow people who are meditatingto see their progress as feedback. This can be helpful for people whohave been asked to meditate as part of cognitive behavioural therapy, orto bring down blood pressure, or manage chronic pain. For example,Chenelle wants to participate in a meditation program at home, but herhome is crowded/noisy/not conducive to mindfulness. To overcome thisenvironmental obstacle, Chenelle dons a pair of VR goggles. The gogglesprovide a virtual meditation environment in which the area surroundingChenelle is free of distractions. This VR environment can be mappedusing a device which contains sensors for mapping a 3D environment, suchas the Google Tango phone. Chenelle can participate in either a walkingor a sitting meditation practice with the distracting elements of theenvironment she's in blocked out. As Chenelle practices her meditation,her EEG state is being monitored. Chenelle can visualize her EEG stateduring the meditation practice; it can be presented like a musicvisualizer—a series of peaks and troughs can become visible travellingtowards her corresponding to her mental state. This can allow Chenelleto see how her meditation is progressing, i.e. whether she is meetingher meditation goals in terms of relaxation, etc. Chenelle can thenoptimize her meditation practice to meet specific goals by modifying herbreathing or some other variables to create a different outcome duringthe meditation practice.

Embodiments described herein translates EEG data, heart rate and pulsedetection, and eye-tracking to generate feedback outputs like real timedynamic changes to the virtual environment (such as a change in music orambient sound, or shifts in light, transparency, or opacity, ortopography), while also creating data for reports that wearers could optinto or share with friends and supporters.

Application: Therapeutic—Creating a Virtual Meditation Group Using VRTechnologies

In a globalized world, it's easier than it ever was before to findlike-minded individuals and spend time with them online. But it's rareto find a digital-friendly meditation studio, or people to meditate withover long distances. This system aims to change that, by creatingvirtual meditation spaces where people from all walks of life can dropin for a re-focusing session. For example, Grant wants to participate ina group meditation session, but there are no meditation groups meetingnear him. Grant puts on a pair of VR goggles which include EEG-sensingtechnology. The goggles are connected to a computer network thatcontains a virtual “meditation salon”, much like a virtual chat room.Other participants who want to partake in the collective virtualmeditation experience can “dial into” the salon. Each person who isengaging in the shared virtual experience is represented in the“meditation salon” by an avatar. The meditation group is guided by aleader who issues instructions that the rest of the group follows. Asthe group proceeds with their meditation session, their EEG data iscollected. As the meditation session proceeds, each member of the groupcan decide whether to keep their eyes closed or open. If they decide toopen their eyes, they are able to look at other member of the group. TheEEG readings from other members of the group affects the appearance oftheir avatars. e.g. if the person is relaxed, their avatars can glowblue; if they are tense, their avatars would glow red. The individualparticipants can look at their own avatars to determine how they areproceeding with the meditation session; their own avatars will changecolour like those in the rest of the group. Emotions can also bedisplayed on the avatars; i.e. the words “ANGRY” or “ANXIOUS” can appearon their faces. Guided meditations can be programmed so thatinstructions issued by the group leader are interpreted visually. i.e.the meditation leader says, “Imagine a beam of bright white lightemanating from the top of your heat. This light pulses in time with therhythm of your breath. This visual metaphor can be translated literallyonto the avatars in the meditation group; users can see beams of lightemanating from the tops of the heads of the other participants in thegroup.

Embodiments described herein can translate EEG data along with motiondetection, eye-tracking, and other physical and brain-state data intodynamic real time experiences in a virtual environment, such as textalerts, shifts in colour, changes in music, and changes in playerrelations as examples of feedback.

Application: Therapeutic, Entertainment—Caring for a Virtual Pet in a VREnvironment

FIG. 11, the system provides a therapeutic environment for people thatis also entertaining. For example, Danielle owns a virtual pet that shetakes care of in a virtual environment. She puts on her VR goggles tovisualize the pet in a specially-constructed 3D environment. The petresponds not only to Danielle's actions (i.e. feeding it; entertainingit, nurturing it) but also responds to her emotional state as differentforms of feedback. While Danielle cares for her virtual pet, she wearsan EEG-detecting device. The device measures her brainwaves and changesthe characteristics of the virtual pet accordingly to provide visualfeedback. For example, the pet changes colour from green to red whenDanielle is upset; it changes back from red to green when she enters arelaxed state. Alternatively, the pet can change its own behaviour:irritated, relaxed, angry, etc. The pet can be used as amindfulness/meditation aide—Danielle tries to get her pet to changecolour to a certain state to match whatever mindfulness goals she isaiming towards. Danielle can also enter a virtual “pet club” to interactwith other virtual pets and their owners. These owners/pets appear in anetworked, constructed virtual world. Danielle can interact with otherpets and their owners and can change the emotional/physical states ofthe pets. If other pet owners in the virtual space are wearingEEG-sensing devices, their mind state is visible to Danielle as variousmanifestations on their avatars—changing colour to red when distressed;to green when relaxed, etc.

Application: Data Gathering—Using Immersive VR Environments to TestAdvertising Effectiveness

Most market research depends on answering survey questions or performinga quiz. Embodiments described herein may provide real time brain statedata to market researchers and other data gatherers about how subjectsfeel about specific stimuli. This data is a complement of physiologicaldata to existing qualitative data in research settings, whether they aremarket-driven, sociological, or purely scientific. For example, Jon isin charge of running a focus group to determine the effectiveness of aTV ad. Jon engages with a focus group audience who are equipped with VRgoggles. These goggles contain EEG-reading hardware. In a purpose-builtVR environment, Jon instructs the focus group subjects to watch an adfor testing. As the participants watch the ad, Jon is able to visualizetheir emotional responses in real time as a stream of data from the EEGreaders on the participants. For example, as Jon watches Barb—who ispart of the focus group audience—she can see that she is being irritatedby the ad because her avatar in the VR environment has turned red.Similarly, Jon can see that Kevin—another focus group participant—isenjoying the ad because the colour of his avatar has shifted to a greenhue. Jon can show multiple versions of the same commercial to the focusgroup and have the ability to detect their real time reactionsinstantaneously.

Embodiments described herein may take data from EEG sensors andtranslates it into feedback outputs like data metrics reports onattentional bandwidth among users. It can also help marketers and othersdetermine how to tweak media in real time based on real time brain-statechanges.

Application: Therapeutic, Entertainment, Design—Using Immersive VREnvironments to Facilitate Engineering Testing

High-pressure systems like nuclear reactors, hydro-electric dams, andother pieces of infrastructure can seem quite daunting to the people whoare training in how to maintain them. Embodiments described herein mayprovide a dynamic, responsive virtual environment that helps trainprofessionals and offers them feedback on their stress levels duringtraining exercises. Using this data, professionals can examine theirweak spots and learn exactly where they should be focusing whileperforming these high-pressure tasks.

For example, Bill works for a hydroelectric utility. He is tasked withretrofitting one of the turbines at the hydro plant he works at. He andthe design team enter a VR environment corresponding to the environmentinside that turbine. Some of the members of this team are across thecountry from Bill and his co-workers, so meeting in a virtualenvironment makes the most sense. He and the team can try variousengineering solutions to solve the problem at hand, or investigatedifferent design solutions from available contractors and vendors. Asthey sketch out different solutions, these solutions appear inside thevirtual world. Every so often EEG-reading devices in the VR headsets cantake the emotional “temperature” of the participants to see how wellthey are performing together.

Embodiments described herein can translate inputs like EEG data,motion-detection, eye-tracking, and other data into a dynamic virtualenvironment that reflects changes in infrastructure design or devicedesign. It can also indicate how different users react to specificdesigns, such as a flash of recognition, disgust, or the sense a givendesign being a mistake.

Application: Entrainment, Medical—Using Immersive VR and EEG to PracticeHigh-Tension Operations

Many training exercises can prove too dry, meaning that the peopletaking them aren't truly prepared. Embodiments described herein mayprovide a dynamic, responsive virtual environment that helps trainprofessionals and offers them feedback on their stress levels duringtraining exercises. For example, Kim is a surgeon. Kim is going toperform a very difficult operation, and needs to practise beforeactually getting into the theatre with his patient. Kim wears a wearablecomputing device of the present invention while watching a simulation ofthe surgery in question. Kim downloads a simulation of the surgery anduses the system to develop skills at performing difficult parts of thesurgery. He learns the rhythm of the surgery, as well as how to handlesurprises like losses in blood pressure or sudden changes in heart rate.The system generates a report for Kim at the end of each session to helphim see when he felt most anxious, and which parts of the surgery hemight feel most anxious about. The report also tells him when his mindwanders or when he's distracted during the surgery simulation. When Kimactually performs the surgery in real life, he feels a great deal moreconfident and calmer. The surgery is successful.

Application: Entrainment, Military—Using Immersive VR+EEG to PracticeActivities

The system may provide a dynamic, responsive virtual environment thathelps train professionals and offers them feedback on their stresslevels during training exercises. For example, Joan works as part of abomb disposal unit. Before she ships out, she has to log several hoursof practise defusing bombs and other explosive devices in a variety ofenvironments. Joan wears a wearable computing device of the presentinvention to practise working with multiple types of explosives. Shedownloads a suite of environments and challenges that represent what shewill experience overseas as part of her unit. The environments arerendered in her VR unit, while the system reads her brain state as sheperforms the task. As the system reads her brain state, it notes whenshe feels anxious or perceives herself to have made a mistake (ERN). Thesystem generates a report after each session that shows her what shefeels most anxious about, and when she was distracted. Over time, Joangrows more confident.

Application: Data-Gathering, Entertainment—Using Immersive VR as a“Pay-Per-View” Client for Live Events, in Exchange for Brainwave Data

For people who want to attend events “live” and contribute to them (suchas sports broadcasts or political conventions), there is a value intheir brain state data. This system can harness that data and feed itback to others who are in attendance.

For example, Tom is a member of the federal Liberal party. He wants toattend the party's annual convention but is unable to go in person.There is a roving 3-D camera at the event that captures key moments inthe event. Tom is able to “dial into” the camera to access the feed—thusgiving him the experience of actually being in the space. This is nottechnically a computer-generated “virtual” environment, but the AVcontent streaming from the convention constitutes a digital data streamjust as much as virtual content coming from a computer. There is anEEG-reading device “baked into” the VR headset that Tom is using. TheEEG device will collect valuable metrics about the convention—i.e. Tom'sreaction to various speeches; his feelings of positivity or negativitytowards candidates, etc. This data can be used by party leaders, pollingfirms, etc. Tom enters into an agreement with the party that the metricscollected from his EEG be used for whatever purposes they desire inexchange for his ability to access the data stream. This technology canbe extended to other types of live events, e.g. pay-per-view sports orconcerts. In each case, the client would exchange the data collectedfrom his EEG device—plus the application of a possible fee—for the datastream from the event. The live TV stream from the event could be mergedwith other data; for example, if the viewer is watching a UFC fight,stats on competitors could be summoned at will and be merged into thegeneral viewing milieu.

Embodiments described herein may allow users wearing their wearablecomputing device to gather brain-state and reaction data on themselvesbased on EEG monitors, eye-tracking, motion, temperature, and otherresponses and share it with a crowd of people also in attendance at thesame event. This allows researchers to create a portrait of the “mood”of an event, while also allowing audiences to engage together at a moreintimate level.

FIG. 16 shows an instance where one user's 1602 VR environment (asprovided by a VR system or wearable device 1604) may be modified basedon biological signal acquisition of one or more other users 1606. Forexample, the user 1602 may be able to see the reaction of other audiencemembers 1606 to the user's performance through the VR environmentrendered by the wearable computing device worn by the user 1602. Theother users 1606 may also have a wearable device with bio-signal sensorsto provide bio-signal data to a bio-signal acquisition unit 1608 and abio-signal processing pipeline 1610 to determine one or more userstates. The bio-signal acquisition unit 1608 and a bio-signal processingpipeline 1610 may access user profiles 1612 stored in a data storagedevice to determine the user states. The user states may be scored toprovide feedback to the user 1602 as visual representations in the VRenvironment. The other users 1606 may also access the VR environmentusing wearable devices and may have the same or different view from theother user 1606 to determine their user states based on VR eventsrelating to all users 1602, 1606 or VR events generated by user 1602that are observed by users 1606 to trigger a brain state reaction asdescribed herein. A VR controller 1614 may trigger updates in the VRenvironment as feedback and may also trigger VR events to evaluate brainstates of users 1602, 1606 in relation to those VR events. The VRcontroller 1614 may authenticate user or receive additional bio-signaldata from user biometric devices 1616.

The present system and method may be practiced in various embodiments. Asuitably configured computer device, and associated communicationsnetworks, devices, software and firmware may provide a platform forenabling one or more embodiments as described above, such as for examplethe computing device 150 and the other computing device 160. By way ofexample, FIG. 15 shows a schematic view of an implementation of acomputer device 500 that may include a central processing unit (“CPU”)502 connected to a storage unit 504 and to a random access memory 506.The CPU 502 may process an operating system 501, application program503, and data 523. The operating system 501, application program 503,and data 523 may be stored in storage unit 504 and loaded into memory506, as may be required. Computer device 500 may further include agraphics processing unit (GPU) 522 which is operatively connected to CPU502 and to memory 506 to offload intensive image processing calculationsfrom CPU 502 and run these calculations in parallel with CPU 502. Anoperator 507 may interact with the computer device 500 using astereoscopic video display 508 (or 2D display) connected by a videointerface 505, and various input/output devices such as a keyboard 510,mouse 512, and disk drive or solid state drive 514 connected by an I/Ointerface 509. Other types of input/output devices may also beinterfaced with the computer device 500, such as game controllers,including various types of gamepads, gesture controllers, and motiondetectors or analyzers. The display 508 may be integrated into thecomputer device 500. Biological signal sensors 540 may also be connectedto the computer device 500 through I/O interface 509 and may beactivated/deactivated or otherwise triggered through the I/O interface509. Each biological signal sensor 540 may operate independently or maybe linked with other biological signal sensor(s) 540, or linked to othercomputing devices. Each biological signal sensor 540 may transmit datato the CPU 502 through the I/O interface 509. In known manner, the mouse512 may be configured to control movement of a cursor in the videodisplay 508, and to operate various graphical user interface (GUI)controls appearing in the video display 508 with a mouse button. Thedisk drive or solid state drive 514 may be configured to accept computerreadable media 516. The computer device 500 may form part of a networkvia a network interface 511, allowing the computer device 500 tocommunicate with other suitably configured data processing systems (notshown). The application program 503 may include instructions toimplement aspects processes described herein, including trainingprocesses that may involve receiving and processing manual input orbio-signal data or a combination thereof. Further, the applicationprogram 503 may include instructions to provide training feedback to theuser based on a combination of user state score and a performance score,or other training feedback in various example embodiments.

It will be appreciated that any module or component exemplified hereinthat executes instructions may include or otherwise have access tocomputer readable media such as storage media, computer storage media,or data storage devices (removable and/or non-removable) such as, forexample, magnetic disks, optical disks, tape, and other forms ofcomputer readable media. Computer storage media may include volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Examples of computer storage media include RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks(DVD), blue-ray disks, or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by an application, module, or both. Any suchcomputer storage media may be part of the mobile device, trackingmodule, object tracking application, etc., or accessible or connectablethereto. Any application or module herein described may be implementedusing computer readable/executable instructions that may be stored orotherwise held by such computer readable media.

Thus, alterations, modifications and variations can be effected to theparticular embodiments by those of skill in the art without departingfrom the scope of this disclosure, which is defined solely by the claimsappended hereto.

In further aspects, the disclosure provides systems, devices, methods,and computer programming products, including non-transientmachine-readable instruction sets, for use in implementing such methodsand enabling the functionality described previously.

Although the disclosure has been described and illustrated in exemplaryforms with a certain degree of particularity, it is noted that thedescription and illustrations have been made by way of example only.Numerous changes in the details of construction and combination andarrangement of parts and steps may be made. Accordingly, such changesare intended to be included in the invention, the scope of which isdefined by the claims.

Except to the extent explicitly stated or inherent within the processesdescribed, including any optional steps or components thereof, norequired order, sequence, or combination is intended or implied. As willbe will be understood by those skilled in the relevant arts, withrespect to both processes and any systems, devices, etc., describedherein, a wide range of variations is possible, and even advantageous,in various circumstances.

What is claimed is:
 1. A training apparatus comprising: an input deviceand a wearable computing device with a bio-signal sensor and a displayto provide an interactive continuous virtual reality (“VR”) environmentfor a user, the VR environment containing virtual elements, thebio-signal sensor receives bio-signal data from the user, the bio-signalsensor comprising a brainwave sensor; the computing device having or incommunication with a processor configured to: as part of the interactivecontinuous VR environment, present content on the display where thecontent includes the virtual elements and has a VR event occurringwithin the interactive continuous VR environment, the VR event havingone or more changes on the virtual elements in the VR environment, theVR event having desired user states, and desired effects; continuouslyreceive user manual inputs from the input device for user interactionwith the virtual elements in the interactive continuous VR environmentincluding during the VR event; continuously receive the bio-signal dataof the user from the bio-signal sensor during the VR event; process thebio-signal data to determine user states of the user, including brainstates, during the VR event, the user states are processed using a userprofile stored in a data storage device accessible by the processor andthe user states include brain states; determine a user state score bycomparing the user states of the user to the desired user states duringthe course of the VR event; determine a performance score by comparingthe effects to the desired effects during the course of the VR event;and provide feedback to the user wherein the feedback is based on acombination of the user state score and the performance score, whereinthe user manual inputs are continuously received independent of the userstate score.
 2. The apparatus of claim 1 where the wearable computingdevice further comprises an inertial sensor and the bio-signal sensorfurther comprises a facial bio-signal sensor, wherein the facial sensorincludes an EOG sensor, and where the bio-signal data further comprisesdata from the facial bio-signal sensor and the inertial sensor; thecomputing device is further configured to: receive the bio-signal datafrom the EOG sensor and the inertial sensor for a user head and eye gazedirection wherein the user states further comprises the user head andeye gaze direction, and the desired user states further comprises adesired user head and eye gaze direction.
 3. The apparatus of claim 1wherein the brain states comprises one or more of ability of operator tolearn; prediction error; and emotional state leading to impairedthinking.
 4. The apparatus of claim 1 further comprising presenting theuser state score and the performance score synchronized with the contentand the VR event to assist the user to better attain the desired userstates and desired manual inputs on the input device.
 5. The apparatusof claim 4 where the computing device is further configured to: revisethe content in response to the feedback provided to the user where theuser is further trained on the revised content.
 6. The apparatus ofclaim 4 wherein the user state score further comprises failure brainstates.
 7. The apparatus of claim 1 where the display is a stereoscopicdisplay.
 8. The apparatus of claim 1 further comprising a second displayfor presenting the content and a visual representation of the userstates and manual inputs of the user in real time.
 9. The apparatus ofclaim 1 wherein the computing device having or in communication with theprocessor is further configured to: provide real time or near real timefeedback to the user during the presentation of the content.
 10. Theapparatus of claim 1, wherein the VR event is associated with event timedata and a portion of bio-signal data is associated with bio-signal timedata corresponding to the event time data, wherein the processor isconfigured to identify a portion of the bio-signal data based on theevent time data and process the portion of the bio-signal data todetermine the user states during the VR event, the bio-signal time datasynchronized to the event time data.
 11. The apparatus of claim 1,wherein the VR event is associated with event time data and thebio-signal data is associated with bio-signal time data, and wherein theprocessor is configured to identify a time interval based on an expectedresponse time for the VR event and the event time data, identify aportion of the bio-signal data based on the time interval and thebio-signal time data, and process the portion of the bio-signal data todetermine the user states during the VR event, the bio-signal time datasynchronized to the event time data.
 12. A training method implementedusing an input device and a wearable computing device having or incommunication with a processor, a bio-signal sensor and a display toprovide an interactive continuous virtual reality (“VR”) environment fora user, the VR environment containing a plurality of virtual elements,the bio-signal sensor receives bio-signal data from the user, thebio-signal sensor comprising a brainwave sensor; the training methodcomprising: as part of the interactive continuous VR environment,presenting content on the display where the content has a VR eventoccurring within the interactive continuous VR environment, the VR eventhaving one or more changes on at least a portion of the plurality ofvirtual elements in the VR environment, the VR event having desired userstates, and desired effects; continuously receiving user manual inputsfrom the input device for user interaction with the virtual elements inthe interactive continuous VR environment including during the VR event;continuously receiving the bio-signal data of the user from thebio-signal sensor during the VR event; processing the bio-signal data todetermine user states of the user, including brain states, during the VRevent, the user states are processed using a user profile stored in adata storage device accessible by the processor and the user statesinclude brain states; determining a user state score by comparing theuser states of the user to the desired user states during the course ofthe VR event; determining a performance score by comparing the effectsto the desired effects during the course of the VR event; and providingfeedback to the user wherein the feedback is based on a combination ofthe user state score and the performance score, wherein the user manualinputs are continuously received independent of the user state score.13. The training method of claim 12, where the wearable computing devicefurther comprises an inertial sensor and the bio-signal sensor furthercomprises a facial bio-signal sensor, wherein the facial sensor includesan EOG sensor, and where the bio-signal data further comprises data fromthe facial bio-signal sensor and the inertial sensor; the method furthercomprising receiving the bio-signal data from the EOG sensor and theinertial sensor for a user head and eye gaze direction; wherein the userstates further comprises the user head and eye gaze direction, and thedesired user states further comprises a desired user head and eye gazedirection.
 14. The training method of claim 12, wherein the brain statescomprises one or more of ability of operator to learn; prediction error;and emotional state leading to impaired thinking.
 15. The trainingmethod of claim 12, further comprising post presenting the user statescore and the performance score synchronized with the content and the VRevent to assist the user to better attain the desired user states anddesired manual inputs on the input device.
 16. The training method ofclaim 15, further comprising revising the content in response to thefeedback provided to the user where the user is further trained on therevised content.
 17. The training method of claim 15, wherein the userstate score further comprises failure brain states.
 18. The trainingmethod of claim 12, where the display is a stereoscopic display.
 19. Thetraining method of claim 15, further comprising presenting the contentand the user states and manual inputs of the user in real time on asecond display.
 20. The training method of claim 15, further comprisingproviding real time feedback to the user during the presentation of thecontent.
 21. The apparatus of claim 1, wherein the desired user stateincludes a distribution of desired brain states and wherein thedetermining of the user state score includes determining the deviationof the brain state from the distribution of desired brain states. 22.The apparatus of claim 1, wherein the processor is further configured todynamically update the user profile using the bio-signal data.